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Welcome to Augmented Reality 

Change your perception of reality by understanding this fascinating 
technology. How different is it from Virtual Reality? Read on to find out 
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Notable achievements in this space 

From the earliest attempts at augmentation in the form of 
Experience Theaters to newer milestones like Google Glasses, 
we chronicle some of the pivotal moments in this space 


Different types of Augmented Reality 

Augmented reality does not end at your smartphone. There are many 
more ways to enhance your perception of the world 
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AR: The Limitless Possibilities 

Augmented reality is not just about fun and games. There are some 
substantially beneficial uses for the technology in everything from 
engineering and medicine to sports, education and more 
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Introduction 

“ Reality is merely an illusion, albeit a very persistent one. ” 

- Albert Einstein 

G iven that exquisite premise by Mr. Einstein, consider this for a 
moment: what would happen if you add to that illusion? Would 
the result be more illusory or more real? Before our minds 
implode under the weight of the conundrum, lets get down to 
what we want to talk about - Augmented Reality. 

Augmented reality (AR) is the altering your perception of the world by 
wrapping a layer around reality. A layer of augmentation. They say reality 
is a hard pill to swallow. So how about reality served with a sugar coating 
of information? Augmented Reality can help in multiple ways - provide 
critical information about your surroundings, give you your personal babel 
fish, help soldiers survive wars better, help law enforcement officials catch 
criminals, avoid car accidents, train surgeons, and much more. But AR is not 
just about this, it can be fun too! Yep it has limitless possibilities in gaming 
This FastTrack will take you into the world of Augmented Reality on a 
magic carpet ride of sorts. We first begin by introducing you to the key con- 
cepts and also differentiate AR from another closely related concept - Virtual 
Reality. Next we tell you about about the earliest attempts at augmentation 
in the form of Experience Theaters to newer milestones like Google Glasses. 
We also try to chronicle some of the pivotal moments in this space. 

The most notable (and famous) implementation of AR, as of now, are the 
location browsers running on mobiles that help you locate points of interest 
around you. As you read this FastTrack, you’ll be surprised to learn that 
(metaphorically) AR is much larger than what a mobile phone can accom- 
modate. You would learn its various uses and types. 

We do not stop there and we don’t want you to either. We explore the 
significance of AR and how it is set to change the world as we see it. We show 
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you how you have already been using this technology and what makes it 
tick; the various displays and sensors that enable it. 

Should you be interested in not just experiencing AR but also creating 
that same magic, the last chapter will help you take your first steps into 
AR programming. We tell you about the domain knowledge required 
for starting off with each type of AR implementation and we end with an 
introduction to some of the most useful frameworks / toolkits available to 
begin your journey. 

As always, happy reading. FJ 
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Change your perception of reality 
by understanding this fascinating 
technology How different is it from 
Virtual Reality? Read on to find out 


W e live in a world where distinguishing between fact and 

fiction is increasingly becoming difficult. In fact the lines 
are blurring so much that “reality” is a saleable product 
now. Don’t believe us? Look around you - heard of a little 
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concept called Reality TV? An entertainment sham that people buy into 
irrespective of whether there is any semblance of reality in what is being 
watched. But of course we digress. When focusing on reality in technology, 
there are a few occasions where your reality is blurred. Everytime you 
play a game for instance. You get so immersed in the game that your sur- 
roundings dissolve, you almost enter that virtual world and when someone 
calls out for you from the “real” world you return to it with a jolt. Almost 
resultantly, if we might add. There are technologies out there that blur your 
reality at varying levels. Some embellish what you experience while some 
completely immerse you into an alternate reality. Augmented reality is just 
one of those realities at sale. 

What is augmented reality anyway? 

If you’ve been a Digit reader for long, you might already know what the 
word ‘augmented’ means. But just in case you’re not, let us try to break it 
down for you - the word ‘augmented’ means enhanced or increased. Simple 
enough right? The phrase simply means “enhanced reality”. 

But if reality is what it is how does someone enhance or embellish it? By 
using a tool such as your smartphone. The concept of augmented reality 
revolves around adding more information to your world (or let’s say just 
your view of the world). The augmentation is usually done by adding extra 
data to the perception of things around us which are not visible normally. 
Let us say you are walking down the road (let’s assume, in a city where you 
have never been) and you see a fenced and open area but it doesn’t have 
a signboard demarcating the plot. You check on your phone and Google 
maps says that the area is a meditation center. See? How your mobile phone 
enhanced your perception of your world? 

The above was a mere example to show how everyday tools help us get 
more out of our reality. But before you grasp the above example as one of 
augmented reality, it’s best we tell you that Google maps does not really 
come under the umbrella of augmented reality. 

Imagine again, you go to Shanghai, a well known city in China. What 
you are looking for is a vegetarian restaurant but since everything is 
written in Chinese which you do not understand, you are unable to figure 
out where to go. You take out your smartphone, connect to the internet 
and launch an app. This app looks through the rear camera and shows 
you what it sees except there’s more than just the actual scene. You point 
it to a building, and it shows a graphical callout that says “hotel” next to 
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Figure 1: An application of augmented reality 



the building, you point it to a different building and it says it’s a non-veg 
restaurant. Further down the road you see another building marked as a 
“theater”. Now, that is what is augmented reality. Such an app would really 
be enhancing your perception of reality; it would add information to your 
reality - it would augment reality. 

Another example for visual augmentation would be glasses which tell 
your the direction in which you are looking. If you are looking towards east, it 
shows on screen what you are seeing (of course) and in addition, tells you that 

you are looking in 
the ‘east’ direc- 
tion (if that is the 
direction you were 
facing). This infor- 
mation is displayed 
on the transparent 
glass. You turn 
right by 90 degrees 
and it would say 
that you are now 
facing south. Add 
‘night vision’ view 
Figure 2: Augmented reality is not a distant future anymore to those glasses and 
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you have a tool that can be used on a night safari (or Osama assassination 
raid) without losing track of direction! In this example, we want to highlight 
the fact that augmented reality is not dependent on an internet connection. 
Such a device would work via an inbuilt electronic compass. You’ll find out 
more about exactly how, in the later chapters. 

Mediated reality 

Just like augmented reality is about enhancing the perception of the world 
around us by adding more information to it, a mediated reality is about 
mediating it. Mediation means “coming in between two things to affect an 
agreement”. Here the two things are you and “the reality”, well, the ‘real’ 
reality for that matter (hence we write it as the reality)! 

If you’re wondering “Isn’t augmented reality the same a mediated reality? 
After all augmentation too comes in between us and the reality!” The answer 
is both yes and no. Actually the question is akin to “Isn’t Apple and Fruit 
the same? After all they both grow on trees!”. Without further ado, let us 
see the actual difference. 

Mediated reality is a way to alter your perception of your surroundings 
by adding or removing information from it. Removing is the keyword here. 
While augmented reality is achieved by adding information to the percep- 
tion of our surroundings, removing information from surrounding would 
qualify for mediation as well. 

Virtual Reality 

Much before augmentation was linked to reality, virtualization was. Virtual 
reality was a term which has been flashed a lot in recent past. Geeks who 
love to watch TV channels of the likes of Discovery and National Geographic 
would have been hearing the term for quite some time now. However, if 
you were having a vacation on Mars all this while, it would be great to get 
informed about virtual reality before we proceed further. 

The term ‘virtual reality’ is easily one of the most self-contradicting 
terms ever coined - an oxymoron, if you will - though it is simple enough 
to understand. Virtual reality means a ‘reality’ which ‘imaginary’. But 
understand that this is not just imagination or processes that happen in 
the mind. It is sensory perception that is induced by tools from the real 
world. The term refers to simulation of reality in a way that it looks and 
feels real but is not. Science is pushing boundaries everyday to cover the 
feeling part well enough. 
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Virtual reality is a simulation (the virtual part) of some part of real 
world (the reality part). In most cases known, the simulation is done in 
either of two ways: 

1 The subject (person) is sent to a room with giant displays. The picture 
being displayed on the walls of the screens all around is calibrated with 
the position where the person is located. This makes the subject see the 
simulation as if he is watching it for real. The acoustics system is synced 
well with the environment as well. 

2 The subj ect is given wearable gadgets which are synced with each other 
and allow him to watch an ongoing simulation (video and audio) as if he 
is watching reality. The wearables also include gloves fitted with sen- 
sors to detect movements. Neck and body movements are also detected 
using other wired or wireless methods. These sensors send signals to 
the controller which then calculates the scene to be shown. The wearable 
devices also send feedback to the subject to make his experience more 



Figure 3: The two ways of doing VR 


real. For instance vibration could be induced on the subjects seat if the 
scene being viewed is that of a jeep going through a Safari. 

Virtual reality is used in a lot of applications such as preliminary training 
exercises (e.g. for pilots and military) and entertainment. One of the lesser 
known usage of VR is for therapeutic purposes on psychological levels. 

Virtual Reality and Augmented Reality - 
similarities 

It might just seem that virtual reality is very different from augmented 
reality. After all, they look so different. Virtual reality is all about making 
you believe that you are in a place where you actually aren’t, experiencing 
sensations that aren’t there. 
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Take for an instance a 3D game that you play on your PC (the likes 
of Crysis, Need for Speed etc.) You could call some of them to have an 
immersive gameplay. Bring in a large display (say a 42-inch display) and 
you can get really lost into the world. In no time you start feeling that every 
single move in the game is actually happening with you in the real world. 
Every bullet you shoot and every turn you take feels like happening in real 
world though none are real. You start living in a reality that does not exist. 
That is the effect which scientists all around the world have been trying to 
enhance and as scientific research goes forward, a few things start coming 
to everyday technology. 

A very popular game, Prototype’s storyline was set in New York city. 
Locations, buildings, roads - all were designed into the game. During the 
gameplay, you run through the streets of the city, climb buildings. Let us 
assume you are not doing the gameplay on a single screen but a room 
with all walls as displays which make you believe that you are actually 
in the gameplay. Wouldn’t that be really a great gameplay? You would be 
totally immersed. 

Imagine an extension of this scenario. What if all those people in the game 
were real? By real we mean graphically generated models of real people right 
down to the clothes they were wearing. What if each one of those people 
were similarly plugged into this virtual universe, through virtual reality 
rooms of their own? The traffic could be created in a way to mimic actual 
traffic conditions on that day. So it the weather. Such a game wouldn’t be so 
immersive that you would actually forget who you are and where you are? 

Would you call that game a game? Or would you call it an application 
built around the concepts of ‘augmented reality’? It’s both. In a way Virtual 
Reality is a higher level of augmentation of reality - it is the peak. In the 
realms of Virtual Reality, it’s not just the information that is added to reality 
but also control of that virtualised environment. 

One of the main uses of virtual reality in the times we live is simulation 
programs that help train people. For instance, training Pilots. Allowing 
a newbie to take control of a fighter jet is extremely risky to both life and 
property and thus, flight simulation is one of the major uses of VR. For a 
VR flight simulator what could be the factors involved? Well, we’ll need a 
3D map of the reality and load it on a computer. We’ll need to simulate the 
Aircraft’s cockpit right down to the last knob and dial. The sounds that the 
plane produces right down to the beep, squeak and grid. We would need 
a pilot’s seat and so on. This virtual environment will even need to react 
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to what is happening for e.g. the seat would need to tilt as the trainee pilot 
banks the plane and so on. 

Go over the scenario and you’ll realise that all that happened was aug- 
mentation. We took conditions from the real world such as altitude and 
wind- speed related information and plugged them into the simulation. 
We enhanced the experience by providing the right type of seat. We added 
the change in view as the plane would tilt. Virtual reality might not seem 
all that different. 

Virtual Reality and Augmented Reality - difference 

All lines between virtual reality and augmented reality are not blurred and 
hazy, There are certain stark differences: 

1 Virtual reality can be built around places that do not exist. While we gave 
an example of Prototype which has been set in Manhattan, there are a 
plenty of games which are set in totally unreal locations and there is no 
way you could connect those places to reality. On the other hand, AR 
is not connected to non-existent worlds. All places on which AR would 
work (both theoretically and practically) have to exist. 

2 Virtual reality requires certain setup which allows the experience to be 
immersive. The concept of VR revolves around building an experience 
so immersive that the subject (okay, we all hate being subjects, makes 
us sound like lab rats, but that is how you would find it written in a lot 
of literature concerning VR) forgets that he is not in the real world and 
perceives the virtual world as reality instead. Setup usually needs tools 
built for this purpose such as special glasses, large screens and other 
wearable gadgets that detect your movements and predict what you are 
trying to do in the virtual world. AR does not need all that. The concept of 
AR revolves around enhancing the reality that already exists by adding 
meaningful information to it. 

3 Usage of AR almost always requires you to be present at the place where 
surroundings have to be to augmented. 

One of the simpler ways to differentiate between VR and AR is that 
VR is about ‘experiencing’ a place whether or not it is real. AR is about 
‘enhancing’ the perspective of a place by adding more information to it. 
All other differences creep in due to the differences in main idea around 
which they revolve. TJ 
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moments in this space 
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A ugmented Reality dates back to the 1950s when Morton Heilig, 
Father of Virtual Reality, invented the Sensor ama Stimulator. 
He dubbed it the “Experience Theater”. The idea behind it was 
to simulate an experience with visuals, sound, vibration, and 
smell, without actually being subjected to the “possible hazards of particular 
situations”. It was designed as a cinematic experience: it blew wind at you, 
vibrated the seat you sat on, played sounds to your eyes and projected a 
form of a stereoscopic 3D environment to the front and sides of your head; 
kind of like those 5D movies that we watch in theaters today. However, 
it was a mechanical 
device shaped like 
an arcade machine 
from the 80s. Oddly 
enough in retro- 
spect, the project 
didn’t get financial 
backing. The first 
augmented reality 
head-mounted dis- 
play system was 
invented by Ivan 
Sutherland in 1968. 

The device was too 
heavy-to-handle 
and was hung sus- 
pended from the 
ceiling and very 
appropriately called 
The Sword of Damo- 
cles. While its graphical prowess was fairly limited, it was the first step to 
usable-AR. It was only in the 1990s that the tech was finally christened, 
when a Boeing researcher Tom Caudell coined the term “augmented reality”. 
Julie Martin created a theatrical production in Australia called “Dancing 
In Cyberspace”, which used Silicon Graphics computers to let the dancers 
and acrobats manipulate virtual objects in real time projected into the same 
performance space. 

For most of the 20th century, AR remained a toy for the scientists. Which 
meant that the common man didn’t even know it existed. Bulky equipment 



Sensorama Stimulator 
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with expensive software kept far from the consumer; until Hirokazu Kato 
of Nara Institute of Science and Technology, Japan released the ARToolKit 
to the open source community. The ARToolKit was a platform for creating 
augmented reality applications. It used video tracking capabilities and 
virtual object interaction, and provided 3D graphics models that could 
be overlaid on any OS 
platform. Smartphones 
were not in the picture 
at the time; the Flash 
based AR applications 
required a simple hand- 
held device with a camera 
and a stable Internet 
connection. Prof. Bruce 
H. Thomas of the Wear- 
able Computer Lab at 
the University of South 
Australia created the first 
ever Augmented Reality video game, ARQuake. You needed your laptop in 
a backpack, GPS, gyroscopes, a head-mounted display and voila! The world 
around you changed into a game arena, with monsters roaming about in 
your vicinity in real time. You didn’t need a joystick; you could move about 
in the real world and destroy the virtual monsters. The game still exists 
as a prototype to this day and unfortunately, no plans to commercialise it 
exist as of now. 

A few years later in 2008, Augmented Reality apps came to your smart- 
phones, where you could finally enjoy this mindblowing technology. With 
applications like Wikitude and Layar which pioneered the smartphone app 
market, the environment around you was suddenly made “clickable”. These 
AR apps use your smartphone/tablets’ camera, GPS, compass and other 
sensors to overlay information on the camera feed to your device. When 
ARToolkit was ported to Adobe Flash, Augmented Reality was made pos- 
sible through your desktop browser and the webcam. 

In February 2009, at the TED conference, Pattie Maes and Pranav 
Mistry presented their augmented-reality system- SixthSense. Their 
prototype consisted of a pocket projector, a mirror and a camera con- 
tained in a pendant-like device that a user wore around his neck. Since 
the camera is positioned on the user’s chest, it augments everything 
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in sight. SixthSense is a remarkable device; it uses these off the shelf 
components and costs approximately $350 to build (not including the 
computer). The micro-projector and the camera are connected to a mobile 
computing device placed in the user’s pocket. The mini-projector projects 
visual images onto your surroundings- walls, hand, table, etc., enabling 
them to be used as interactive screens. The camera and mirror are used 
to recognize and track a user’s hand gestures, colors and physical objects 
using image-processing based techniques. The software on the computing 
device processes the video stream data captured from the camera using 
simple computer-vision techniques, gathers GPS coordinates and pulls 
up data from the Internet. The user may also wear colored caps on the 
fingers which are used to interact with the projected image; even fingers 
with different colors of nail polish would work. The movements and 
arrangement of these colored markers (also known as fiducials) and hand 
gestures serve as interaction instructions for the projected application 
interfaces. SixthSense also supports multi-touch interaction letting you 
zoom in, zoom out or pan using many of the intuitive hand movements 
that we are now familiar with. The SixthSense prototype implements a 
number of demonstrative applications: 

► The map application lets you navigate a map displayed on a nearby 
surface using hand gestures. 

► The drawing application lets you draw on any surface by tracking the 
fingertip movements of the user’s index fingers. 

► SixthSense also lets you draw symbols in the air using the tip of your 
index finger and recognizes those symbols as interaction instructions. 
Drawing a magnifying glass symbol takes you to the map application or 
drawing an symbol lets you check mail; Draw a circle on your wrist 
and SixthSense projects an analog watch on it. 

► SixthSense recognizes your complex hand gestures. The SixthSense 
system takes photos of the scene you are looking at by detecting the 
’framing’ gesture. You can stop by any surface or wall and flick through 
the photos you’ve taken. 

► The SixthSense system also augments physical objects in your vicinity 
by overlaying more information on them and making them interactive. 
For example, a newspaper can show live video news or dynamic infor- 
mation provided on a regular piece of paper. Touch any object and learn 
so much more about it than appears to the eye. 

For more details on SixthSense log on to: http://dgit.in/Xn9EZw. 
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Pranav Mistry with SixthSense 


Steven Spielberg’s 2002 
movie Minority Report 
envisaged life in 2054, and 
shows the exceptional full- 
fledged Augmented Reality 
at work. For sci-fi fans, the 
pertinent memory is that 
scene in the movie when Tom 
Cruise’s character Chief John 
Anderton strolls through a 
mall while everything adver- 
tised itself by bombarding 
him with personalized mar- 
keting messages. A Guinness 
billboard calls him by his 
name and tells him he could 
use a drink; the Gap store has 
holograms of assistants who 
ask him if he’s enjoying his 
previous purchase; an Amer- 
ican Express advertising dis- 
plays a giant 3D credit card 
embossed with his member- 
ship details and so on; you get 
the idea. This “advertising 
nirvana” is based upon the 
possible existence of retinal 
scanners; also, computer 

monitors and displays are transformed into holograms. With the virtual 
retinal display technology, the image is scanned directly into the retina 
of human eye. The viewer sees what appears to be a conventional display 
floating in space. This fancy technology is far from being a figment of 
your imagination and is currently under development at the University of 
Washington’s Human Interface Technology Laboratory. 

Augmented reality is changing the way we see the world. Someday you’ll 
be walking down the street with augmented- reality displays, which will 
eventually look much like a normal pair of glasses, informative graphics 
will appear overlaid on your field of view in real time with synced audio. 



SixthSense's Visual Dialpad 
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AR in Hollywood; a scene from Minority Report 


These enhancements will be refreshed continually to reflect the movements 
of your head. This remarkable technology blurs the line between what’s real 
and what’s computer-generated by augmenting what we see, hear, feel and 
smell. In the AR wars, Google is a step ahead of everyone. Google’s Project 
Glass is developing a wearable spectacles like device that would overlay 
information on the eye’s view and could interact with the Internet via 
natural language voice commands. The device will have Google’s Android 
operating system. The prototype has been worn by Sergey Brin, co-founder 
of Google, to several occasions. He demoed the glasses at the Google I/O 



Sergey Brin with Google Glasses 
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presentation in June 2012 where skydivers, abseilers, and mountain bikers 
wore the glasses and live streamed their point of view to a Google+ Hangout, 
which was also shown live at the Google I/O. 

Google Glasses have also been compared to Steve Mann’s EyeTap. An 
EyeTap is also a type of wearable computing gear for one eye invented by 
Steve Mann in 1980s. The design allows the user’s eye to operate as both 
a monitor and a camera. The first prototype consisted of a computer in a 
backpack wired up to a camera and its viewfinder which in turn was jerry- 
rigged to a helmet. With time, the technology evolved and the size of the 
device shrinked. The current version consists of an eyepiece that mediates the 
reality, a keypad with which 
the user can interface with 
the EyeTap and a small 
processor which can be 
attached to almost any- 
thing, even a WiFi device 
for pulling up data from 
the Internet. Delving into 
the technology behind this 
amazing gear, we find that 
an EyeTap uses something 
called a “beam splitter” to 
send the same image to both 
the eye and a camera. The 
camera then digitizes that 
reflected image and sends 
it to a computer, which 
processes it. The result of 
all this processing is data 
that is sent to a projector that will then be overlayed over the scene that 
you are viewing 

Interactive eyemaker Vuzix will also reveal their competitor to Google 
Glasses- Smart Glass M100, in CES 2013. The Vuzix eyepatch will have 
Android IceCream Sandwich and a 720p camera along with Wi-Fi, Bluetooth 
and head-tracking sensors. You can take stills, capture video (with playback 
and picture viewing options), answer the phone with a visual address book, 
read emails and texts and run various AR applications. This “modernized 
monocle” is supposedly iOS and Android compatible. Vuzix, which spe- 



Steve Mann with EyeTap 
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cializes in technofancy optics, 
created the first video eye- 
wear to support stereoscopic 
3D for the PlayStation 3 and 
Xbox 360. It also created other 
Augmented Reality eyewear, 
including theWrap 920AR, 

Raptyr and STAR 1200. 

Not lagging behind in 
the race, Microsoft has also 
patented Augmented Reality 
glasses for “live events”. 

Here, “live event” is a broad term encompassing sports, singing and 
other things that are happening around you, including action replays and 
lyrics. Microsoft has recently demonstrated live language translation to 
and from Chinese. These AR glasses would enhance the stream of video 
by overlaying informative graphics. The device is expected to cover a wide 
market with its “live event” idea. According to Patently Apple, Apple too 
has got patents for its own retinized version of Google Glasses, granted 
in February 2010, which would work with a “handheld communication 

device”. Although the 
sketch in the patent is 
not very impressive, 
its an established fact 
that Apple never disap- 
points with its design. 

With glasses and 
head-mounted displays 
in the making, compa- 
nies are also focussing 
on developing AR con- 
tact lenses. These lenses have embedded in them integrated circuits, LEDs 
and antenna for wireless communication. They let the wearer focus on 
two fields of view, the close-to-the-eye AR images and distant real world 
objects at the same time. Innovega, the company that makes them is trying 
to emulate display screens akin to what Arnold Schwarzenegger’s character 
saw in the movie Terminator. These lenses, called iOptik are very much in 
demand by the US military. Now we might have agents like Ethan Hunt 



Apple's Sketch for AR glasses 
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from Mission Impossible 
4: Ghost Protocol, wearing 
these AR lenses and pulling 
up information about eve- 
ryone in sight. News on the 
block is that even Microsoft 
is trying to build AR lenses 
with help from the Univer- 
sity of Kansas. 

Apart from the wearable 
computing devices that use 
Augmented Reality, the tech- 
nology is being exploited in 
many other areas. One of the 
great examples is the AR 

sandbox, created by researchers at UC Davis University of California, 
which overlays a virtual, colored topographic map on a sand surface in 
real-time. As you change the topography of sand, make riverbeds and 
mountains, the 3D visual image adheres to the changes in terrain configura- 
tion. You can actually see the water flowing in the channel and fill up the 
basin. The image is overlaid by a Kinect 3D camera which is connected to 
a projector that overlays virtual information on the sandbox. The system 
is powered by an Intel Core i7 CPU and a Geforce 580 graphics processing 



iOptik Contact Lenses 
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unit. Such AR systems can be used for educational purposes, for example 
for projects in science exhibitions. 

Another cool device based on this remarkable technology is the Aug- 
mented Reality glove for your ipad. Ironically, now you can operate your 
ipad, a touchscreen device, without touching it. With this AR glove called 
T(ether) you can create virtual environments on the tablet. It lets you control 
ipad objects-pick them up, drop them down, push them aside or even create 
a new object by tracing the shape’s outline- with simple hand gestures. 
T(ether) can be used by a number of people to collaboratively work on 
the same virtual environment for the editing and animation of 3D virtual 
objects. By using Vicon motion capture cameras, the position and orientation 

of the tables, user’s 
heads and hands, 
are tracked and 
spatially annotated 
in real-time. Watch 
the video on the fol- 
lowing link to see 
how it works (http:// 
dgit. in/Uo 6x2 P). 
With man still 
trying to per- 
fect the above 
mentioned AR 

devices, Augmented Reality has managed to find a foothold in the world 
of apps for your smartphones and tablets. It will be a while before you 
own a pair of AR glasses or an HMD like the SixthSense, but you can taste 
this sweet fruit via your smartphones and tablets. iPhones and Android 
devices have a vast market of AR apps for almost every aspect of human 
indulgence, from finding restaurants to recognizing landmarks and looking 
up satellites in the sky. More information on various popular AR apps is 
given in chapter 10 of this booklet. These AR apps use your device’s GPS 
for looking up geotagged information, use your camera to receive relevant 
information of the world around you, and so much more. 

And now AR is even making its way into cars. It builds on the “heads 
up display” (HUD) which is a transparent display that presents data to 
the user without him having to look away from the road. This technology 
was developed for military aviation for the convenience of fighter pilots 
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and began to make its way into high-end cars. These HUDs can be used 
to project directions or speed and other relevant information onto the 
windscreen in the driver’s line of sight so the driver need not take his eyes 
off the road. While all this sounds very fascinating, augmented reality 
could allow these displays to get even better. Various car manufacturing 
companies such as BMW, Toyota, Mercedes and GM have shown off their 
prototypes of looking glass integrated with AR technology. The Mercedes 
Dice prototype conjures up information about passing places of interest 
and shows icons of friends driving past the car on the windshield with their 
social network status. Toyota has shown off their version of “enhanced 
car windows” that will allow users to zoom in on places of interest that 
they pass by. The enhanced vision system developed by General Motors 
goes one step ahead and uses an array of sensors and cameras mounted 
inside and outside the vehicle to monitor the on-road conditions. Apart 
from acting as a navigation guide, it helps you drive in foggy conditions and 
generate warnings about possible dangers. The display adjusts according 
to the driver’s head movements. 

Every one of us have different expectations from this technology. The near 
future can be seen but in the long term, this technology might always sur- 
prise you. One interesting perspective is that of the InteraXon Co-Founder 
Ariel Gartern. The company is working on a brain- sensing headband that 
receives your brainwaves 
to monitor concentration 
levels, reduce anxiety, 
improve your memory 
and much more. They 
look forward to a future 
where brainwaves and AR 
work in tandem. The head 
mounted displays and AR 
glasses provide the oppor- 
tunity to collect brainwave 
signals while the user continuously wears the device. This collaboration 
allows the computer system to present contextually aware overlays by noting 
your likes and needs. The system could continuously register and process 
your level of distress throughout your workday and present information 
accordingly. These systems would be “context aware” and would not only 
show relevant information but also present it in the way you want. For 
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example if it senses that you’re sleepy, it would show you the hotels in the 
vicinity to spend the night. The real-time neuro feedback will let it know 
your brain state and try to optimize your experience. This is just one futur- 
istic view; there is so much more to come. The bright side is that developers 
already have a far-sighted view and plans to exploit this technology and 
research is being done to make it a reality. El 
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Augmented reality does not end at 
your smartphone. There are many 
more ways to enhance your perception 
of the world 
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S martphones are probably one of the first objects where us mortals 
got to experience augmented reality; after all they are omnipresent. 
However, with time, as we look around us and observe closely, we 
find that augmented reality is prevalent elsewhere too. 

Let us have a look at the types of AR. Before we begin, understand that 
lines are blurred, broken and twisted. Definitions explained here are not 
written in stone. 


Projection based AR 

Just like anything else which is beyond our reach, projection based AR feels 
more attractive (at least as of now) compared to an AR app you can install 
on your phone. As is obvious by its name, projection based AR functions 
using projection onto objects. What makes it interesting is the wide array 

of possibilities. 

One of the simplest is pro- 
jection of light on a surface. 
Speaking of lights, surfaces 
and AR, did you ever think 
those lines on your fingers 
(which divide each finger into 
three parts) can create 12 but- 
tons? Have a look at the image 
and you would quickly grasp 
what we’re talking about. The 
picture depicts one of the sim- 
plest uses of projection based 
AR where light is fired onto a surface and the interaction is done by touching 
the projected surface with hand. The detection of where the user has touched 
the surface is done by differentiating between an expected (or known) projec- 
tion image and the projection altered by interference of user’s hand. 

One of the widespread uses of projection based AR techniques is non- 
interactive. Projection on objects can be used to create deception about 
the position, orientation and depth of an object. In such a case an object is 
taken into consideration and its structure is studied in depth. The object’s 
distance from the projection is calculated and the projection light sequence 
is then designed carefully to deceive the viewer’s mind. 

The most exciting implementation of projection-based AR is perhaps the 
most exciting one of all uses. We are sure it can appeal to your inner scientist 



Projection-based AR can build you a castle in air, or 
a dialer on hand 


■:tv" 


DIFFERENT TYPES OF AUGMENTED REALITY 


27 


and even the child inside you. Pop over to your PC or Smartphone and watch 
this video :http://dgit.in/U9S4At. It shows Laser Plasma technology in action 
creating 3D images in air. That’s right - 3D images in thin air. This is one of 
those technologies which can not only be used for AR, but for a whole lot 
more. Not only can this piece of tech be used for creating virtual object in 
the air itself, its future developments may make it useful enough for much 
larger deployments. In some cases, it might eliminate the use of special 
gear (such as goggles and head mounted displays) for experiencing AR. 

Recognition based AR 

Recognition based AR focuses on recognition of objects and then provide 
us more information about the object, e.g. when using your mobile phone to 
scan a barcode or QR code, you actually use object recognition technology. 
Fact is, except location based AR systems, all other types do use some type 
of recognition system to detect the type of object over which augmentation 
has to be done. 

Recognition based AR technology has varied uses as well. One of them 
is to detect the object in front of the camera and provide information about 
the object on screen. This is something similar to the AR apps for travel- 
lers (location browsers). However, the difference lies in the fact that the AR 
location browsers usually do not know about the objects that they see while 
recognition based AR apps do. 

A second type of recognition-based AR application is to recognize the 
object and replace it with something else. The applications, once again are 
in abundance and possibilities endless. Some examples are given below: 

1 Simulation of objects in 3D. In this case, printed version of a recog- 
nizable object (such as a card with QR code printed on it, or a picture 
provided by the app printed 
on paper) is shown to the 
camera. This printed ver- 
sion is called “Augmented 
Reality Marker” and acts as 
a reference for the AR app 
running on the system. The 
augmentation app detects 
and recognizes the marker 
and tries to understand the 

distance and orientation of Recognition based AR at work 



tfVrtfrdidit .COfT 


28 


DIFFERENT TYPES OF AUGMENTED REALITY 


the print. Once the recognition is complete, it replaces the marker on 
screen with a 3D version of the corresponding object. This allows the 
user to investigate the object in more detail and from various angles. 
Rotating the marker would rotate the 3D imagery as well. 

2 Yet another famous use of recognition AR tech is - translation of words 
on the fly. In this case, the app reads the words seen by the camera and 
tries to recognize the words using OCR (Optical Character Recognition) 
technology and then replaces the words on screen with their translated 
versions. This can be immensely useful for tourists when travelling to 
places where the locally prevalent language is not known. 

3 Recognition based AR can also be used in education. Markers of more 
than two objects are kept together. The app detects the multiple markers 
and tries to simulate relationships among them. For example, one can 
use printed cards to represent 
atoms (in say chemistry class) 
and based on their mutual dis- 
tance the AR app can show how 
a reaction would take place; 
and that would be just one use 
of AR in education. Detection 
of drawings and sketches by 
more intelligent apps can help 
teach small children, e.g. a 
picture of a giraffe be replaced 
with a living 3D version of a giraffe and children could see how it looks 
in reality and they could interact with the same on a touchscreen! 

4 Recognition of printed versions of 3D objects can help create 3D simu- 
lations of those objects without having to actually build a physical 
model. This can be of great aid to people who constantly work with 
3D applications such as architects and animators. We will talk about 
them in later chapters. 

5 Recognition based AR can be used in projectors to automatically detect a 
projectable surface and project on only the projectable area. The projec- 
tion can be made interactive by using dynamic objects in the surround- 
ings to command the projector. This can eventually be used with the 
projection mapping technique to auto-detect various types of objects 
and send out projection imagery according to the size, distance and 
color of the surface on which projection could be done. 



Augemented Reality for simulating chemical 
formulae 
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With projection based AR, your imagination is the only limit. There is 
a lot of research going on in this field and with time, more and more appli- 
cations would pour in. If you are really excited about how you can create 
something like that of your own, we have tips for you at the end of this book. 
For now, let us see the location based AR. 

Location based AR 

It would be an injustice not to mention this category when talking about 
AR. Location based augmented reality is one of most widely implemented 
applications of AR. The strongest force behind this is the easy availability of 
smartphones and the features that they provide in terms of location detec- 
tion. Location based AR is mostly used to help travellers in their journey. 

Location based AR in most cases is used for AR location browsers 
which help users discover interesting places within their current location. 
This method works by detecting the user’s location and orientation by 
reading data from the mobile’s GPS, digital compass and accelerometer 
and predicting where the user is looking; then adding related information 
on screen about the objects that can be seen from the camera. Wikitude 
is one such application. If you’ve never used the app fear not, this video 
would help you realize how it looks in action: http://dgit.in/WkXxwi There 
are other interesting uses of location based AR which we will talk about 
in the coming chapters. 

Outlining AR 

Though the human eye is known to be the best camera in the world, there 
are limitations. We cannot look at things for too long. We cannot see well in 
low light conditions and sure as anything, your eye cannot see in infrared. 
For such cases, special cameras were built. Augmented reality apps which 
perform outlining use such cameras. Once again, object recognition 
sits behind all that outlining AR can do. Let us begin with a life-saving 
implementation example. 

1 When driving a car on a road in foggy weather, the boundaries of the 
road may not be very visible to the human eye, leading to mishaps. 
Advanced cameras tuned specially to see the surroundings in low light 
conditions can be used to outline the road boundaries within which 
the car should stay. Such a system would prove very useful in avoiding 
accidents. With extra sensors capable of detecting objects around (e.g. 
by using ultrasound) the overall risk of hitting some living object can 
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be minimized as well. The 
technology can help you 
save pedestrian lives as 
well. Outlining people 
crossing the road on a 
HUD (Heads Up Display) 
windscreen can be more 
useful than having a sepa- 
rate infrared video feed. 

2 An AR app known as 110 
stories allows you to see 
the twin towers of world 
trade center in New York 
as an outline. This might a 
tad morbid but it certainly 
is interesting. Outlining 
monuments from history 
can be utilized for many 
purposes the foremost is 
of course education. 

3 Outlining AR can be used by engineers and architects to look at a building 
and study the location of supporting pillars and metal bars. Such an 
application could fetch the building’s construction data from a database 
and provide a mesh/framework view of a building and can prove to be 
an aid in repairing a building or be used for educational purposes. 

Superimposition based AR 

Superimposition based AR provides an ‘alternate’ view of the object in 
concern, either by replacing the entire view with an augmented view of the 
object or by replacing a portion of the object view with an augmented view. 
In this case, once again, object recognition plays a vital role - logically, if 
the application does not know what it is looking at, it most certainly cannot 
replace the original view with an augmented one. 

Depending on what type of view is required, the technology can be used 
for multiple purposes. 

1 Doctors can use the technology to examine the patient from various 
angles in real-time. A live feed from an X-Ray machine can be used to 
superimpose the X-Ray view of the patient's body part on the real image 



Outlining the road can help save you a mishap 



Outlining the buildings from yesteryears is a crea- 
tive use of outlining based AR 
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to provide better understanding of the damage to bones. The application 
can be made to work via a head mounted display or special goggles. In 
other uses, the view can be shown on a screen where the video feed is 
taken from a real camera and X-Ray vision can be imposed on it. 

2 In military applications, superimposition based AR can provide multiple 
views of a target object without showing extra information in text and 
blocking the vision of soldier from other important objects around. If 
you have been shooting enemies via your computer mouse, you’d already 
know how it would appear. Superimposition of infrared view or radioac- 
tive view of an object or an area can help save lives; or win wars! 

3 Superimposition of ancient pictures over real ones can provide interesting 
views of historical places. Broken monuments can come back to life in 
all their original glory. Perhaps different eras complete with landscapes 
can be re-lived with AR. 

4 To allow a tiger or snake near you might be a horrifying experience 
with hazardous consequences, except when superimposition AR is 
used to bring them to you. Placing a person in a location or situation 
which is otherwise dangerous can be can be safely accomplished via 
superimposition AR! 

5 Superimposing a real object with its internal view can be helpful in 
education as well, for instance, to study bone structure. 

Though we have touched some 
of the most important types of 
augmented reality, there are a few 
others which cannot be easily clas- 
sified to fall in one of the above 
said ones. These types use more 
multiple sensors and may produce 
the result in different forms. Such 
systems employ special faculties of 
dedicated devices working in par- 
allel with processing systems with 
algorithms developed especially for being used in AR. As more input and 
output devices converge to more powerful and efficient systems, we would 
come across more types of AR in future. TJ 



Superimposition AR can be used for educa- 
tional purposes as well 
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Augmented reality is not just about 
fun and games. There are some 
substantially beneficial uses for 
the technology in everything from 
engineering and medicine to sports, 
education and more 
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T echnology usually evolves through two pathways - someone 
“builds” a new technology out of necessity to solve a particular 
problem, or someone “discovers” an existing technology and finds 
ways in which it can be used to solve new problems (which the 
original technology may not have been designed for). Augmented Reality 
(AR) is one field that owes its evolution to both aspects equally. As new 
information demands emerged, new ways of adding information to reality 
were found. Each passing day sees an increasing number of applications 
of augmented reality, and its potential seems endless. In this chapter, we 
show you how AR is already modifying our lives in some key areas, and 
what to expect in the coming years. 

Military 

Although, the prime motivator behind all innovations in military technology 
is to make it easier to defeat the enemy and prevent the reverse, several 
technological breakthroughs do find their way into civilian hands. AR is 
one such advancement that has significantly increased the success rate for 
other systems and even military. 

Wait a minute! Chances are that you are wondering when was the last 
time you used AR, and that too as a military application. First person shooter 
(FPS) games, of course. Most FPS games show some really innovative use 
of AR: name of buildings, teammates and weapons as you hover the mouse 
pointer over them, and indicating directions to the next checkpoint without 
ever having to shift your focus from the screen center. 



AR can be very useful in military 
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However the future promises a lot more. Aided with superior tactical 
gear carried by military personnel during an operation, AR can even reveal 
things that are hidden to the naked eye - enemy movements behind physical 
objects, type and range of enemy weapons, points of vulnerability in the 
enemy’s defences, simultaneous tracking of multiple enemy personnel - all 
of which soon prove indispensable to infantry of tomorrow. 

Picture is a screenshot from a game. Please crop the more interesting area 
if needed. Can be used as-is though (text provides grounds for relevance) 

Vehicle Windscreens 

Gaming is so much of an inspiration for AR that this application has also 
materialized out of popular games. Being a smart Digit reader, you would 
have already guessed what genre of games we are talking about: racing. 

When you race cars, bikes, trucks, kayaks (basically, anything under 
the sun) in a game, a lot of information is usually displayed on screen. This 
may include speed (displayed either on a digital or an analog speedometer), 
distance from a vehicle, milestone, or random objects in front of you, a map 
of the path that you are following and visual indications of a sharp turn 
ahead. Depending on the game, your virtual windscreen may also tell your 
where cops are waiting for you and whether or not they are following you 
in a chase. While several developers are working on it, the technology still 
has to see the light of the day in the real world. One simple application of 
would be to send a visual alarm as soon as you near or exceed the speed limit. 

The innovation and possibilities do not end here. If AR is blended with 
MANETs (Mobile Ad-Hoc NETworks), it can be used to provide drivers 
with information slow-moving traffic or nearby accidents. On detecting 
low fuel levels in your gas take, in addition to alerting you, AR may also 
visually guide you to the nearest fuel station. Through visual notifications 
and icons displayed directly on-screen, AR will guide you to your desired 
destination in a new city, without needing you to calculate the ‘next turn 
according to map’ by looking down at an LCD screen. 

We are not going to put any pictures but links to two videos: 

1. GM testing AR with one of their cars: http://www.youtube.com/ 
watchfv =94dg2D -jAhM 

2. BMW’s HUD: http://www. youtube. com/watch?v=b4jpuoGP2C8&feature=endscreen 

HUDs 

HUDs (Heads Up Displays) are spectacles, windscreens, or other surfaces 
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AR in Heads up display 


which allow the viewer to see through them but add extra information over 
(or around) some specific objects. They allow the viewer to get more infor- 
mation about things that he is looking at, without having to shift his focus 
to a secondary panel. Fighter jets are among the most loyal consumers of 
HUDs, as popularized in various movies, such as Top Gun way back in 1986! 

HUDs are one of the most important implementations of AR. Fighter 
planes are built to be used in close-combat situations, where a even the 
slightest mistake can cause heavy losses. Hence it becomes extremely 
important that the human operator flying the plane has access to as much 
information as he can about the things he is seeing without having to shift 
his eyes from the scene up ahead. 

HUDs are being brought into the hands of the common public via spec- 
tacles (it’s gonna take some time though). Remember Google’s Project Glass? 

Please keep the picture as large as possible to show the details 

Medical Sciences 

The fact that AR can provide more information and with minimum distrac- 
tion makes it a prime candidate for the medical sciences. There are already 
apps such as Mole Detective that determine the likelihood of skin cancer 
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by analyzing an image taken using your mobile phone’s camera! Of course 
we wouldn’t vouch for their efficacy at this stage. 

AR has considerable potential in saving both doctors’ and patients’ time. 
AR when combined with current advanced medical equipment can help vis- 
ualize and graphically dis- 
play the medical issue with 
the patient. Special goggles 
can superimpose an x-ray 
image, previously captured 
by an x-ray machine, with 
the real-world view of 
the patient lying on the 
operation table. 

AR is also used in MIS 
(Minimally Invasive Sur- 
gery) where a small camera 
is inserted into the patient’s 
body. One use-case involves using AR to help visualize the brain inside a 
patient’s head. Another involves highlighting the location and extent of 
hairline fractures in a patient’s bones, and recommending ideal next steps 
an orthopedic surgeon. Other uses include visualization of anatomical 
joints in live motion, tracking small surgery tools during an operation 
and simulating the birth of a baby to check for any possible complications. 

Shopping 

Sales are what makes our world go round, and hence developers always 
seek to connect technology to sales. AR may prove to have immense impact 
on retail sales. 

Although the internet has revolutionized shopping to this day, we still 
tend to visit brick- and-mortar outlets on a regular basis. One of the major 
reasons people capable of buying goods on the internet go to physical out- 
lets is to ‘see, touch and feel’ the product before they buy them. Malls can 
leverage the capability of AR to its best by allowing consumers to choose the 
best product for themselves via ‘virtual trial rooms’, which allow shoppers 
to look at themselves in particular outfit or an accessory without having 
to actually wear it. 

The technology works by using a camera (or multiple cameras) to capture 
an image of the shopper, and superimposing the clothing item or accessory 
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AR in virtual trial room 


of the ideal size over the shopper’s body (instead of that of the model used 
in the advertisement). A look at the numbers and you would be able to 
imagine how much profitable AR can prove to retail businesses. Imaginate, 
a company that developed a similar product says 
that the business is worth $600 million if imple- 
mented by today’s brick-and-mortar shopping 
industry. Augmented reality can equally benefit 
web-based retailers. Do watch this video to see 
how AR has already made an entry into the retail 
sphere: http://www.youtube.com/watch?v=NxQZuo6 
pFXJw&feature=player_ embedded 

Tourism 

Tourism is another industry that may can utilize AR to its advantage. Even 
according to Wikipedia / Augmented reality (AR) is a live, direct or indirect, 
view of a physical, real-world environment whose elements are augmented 
by computer-generated sensory input such as sound, video, graphics or GPS 
data.” Including ‘GPS’ with sound and video in AR makes it really valuable 
for travel and tourism business. Let us first take a small step back to try and 
understand the ‘why’ behind the AR- -Tourism relationship. 

Travel is both a luxury and a necessity of modern life. No matter what 
the reason be, when travelling, we are always looking at things or looking 
for something. For example, if you are travelling in Kolkata and you pass 


NOTE 

The picture isa 
snapshot taken from 
the video mentioned 
above. Can be re- 
moved if needed 
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by Victoria Memorial not knowing what that building is and its place in 
history, you would probably say “ah, that looks like a nice colonial palace” 
and stop at that. However, while travelling, we are continually looking out 
for information, let us say the nearest restaurant in a city unknown to you. 
An AR browser application (which helps you browse your locality with AR 
information) could visually guide you to your point of interest. 

From yet another perspective, travel was for long been associated with 
carrying paper maps, books to read during a journey, and paper to take 
notes on. The mighty mobile phone has replaced most of these items. 
Further, the development of smartphones has reduced (if not eliminated) 
the the need to carry multiple ‘information sources’. A single device can 
now encompass a GPS, camera, accelerometer, internet connectivity and 
a compass in our mobile phones. 

AR browsers can guide you step-by-step from point A to point B 
without ever needing to look at a map. However most AR applications 
currently do not ‘understand’ what is in front of you. For instance, cur- 
rent AR applications do not transmit the live video feed sourced through 
a mobile camera to some remote server to tell you what you are looking 
at (such an approach would be very expensive from the computational 
viewpoint and also consume a lot of bandwidth). Instead, information 
about where you are standing and what direction your camera is facing, 
in the form of GPS, compass and gyro-sensor data, is sent to the backend 
server of the app. 

However, these technologies still require several improvements in accu- 
racy and usability. Since the application does not understand what you 
are seeing and depends on GPS, gyro-sensor and compass data, the una- 
vailability of any of these services will degrade the experience of the app 
by a large degree. For example, if the compass in your phone accurately 
calibrated by 90 degrees (i.e., east becomes north and north becomes west), 
then depending on AR app would require you to hold the phone in front of 
you and walk to your left! If the gyro-sensor fails, then the app might just 
tell you to dig the ground to reach your destination. However, such a case 
would be rare. What is more common is losing the GPS signal which would 
make the AR browser totally useless. 

Even with GPS, the information being displayed can be slightly wrong. 
GPS detects your position with an approximate error of anything between 
10 to 20 meters. Such an error can show you wrong information when you 
are at crossroad where multiple roads intersect or in a historical place where 
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AR can be fun as well as useful when travelling 


you are jam packed with points of significance. Also, in most cases, people 
tend to ignore the AR applications and instead rely on mapping services 
such as Google maps to get what they want. Again, it is quite a difficult 
task to keep holding your mobile phone in front of you at all times to learn 
about the place you are in. 

That was all about how AR can be implemented in the tourism industry. 
It’s time we find out what is already out there. 

Multiple applications with AR capabilities exist today, most of which 
work by combining GPS, compass and gyro-sensor information to figure 
out where you stand and what you are looking at. However, these applica- 
tions are still incomplete in terms of data. A major portion of the world still 
needs to be mapped and it is going to take significant amount of time, effort, 
money and computational resources to store and deliver that information. 
One of the most famous AR applications available for iPhone and Android 
is Layar (pronounced ‘layer’) where users can create their own layers filled 
with information of places they like and the appropriate layer can then be 
loaded on the application to assist you using the AR techniques built into 
the app. The by the application creates the augmented reality by deriving 
information from the layer’s data. 
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AR can be used to relive the past! 


Individual users creating layers of information for small places with 
a layer-based approach makes it a lot more easier and helpful, than for a 
single organization to try and map out the whole world. For example, one 
may create a layer containing information of a museum’s exhibits, and 
allow visitors to access this information by simply pointing the camera 
at an object of their interest. It might also be used to ‘re-live the past’. For 
example, if you see a sword in a museum, then AR could make it even more 
interesting by showing it in action — such as presenting the image of a past 
emperor to whom it belonged actually holding the sword! 

Another important use of AR for historical places can be in form of 
historical image superimposition. For example, you visit New York and 
point your camera to Statue of Liberty; an AR app then lets you choose 
images of the Statue and its surroundings at different points in time, and 
then superimposes them on the live image. 

Architecture 

If you have seen either of the movies ‘Prometheus’ or ‘Avatar’, then you already 
have a hint of how AR can help in architecture. AR can help visualize struc- 
tures before they are actually built and simulate their behavior under various 
weather conditions. Not only this, it can also help convert 2D prints into 3D 
visualizations and project time in space in front of you, thus providing a much 
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better understanding of the building 
than what a normal print can convey. 

Yet another way of using AR in 
architecture is to utilize superimposi- 
tion techniques to visualize how an 
architectural change would look in 
reality. Utilizing AR in an architect’s 
workspace can help the architect 
visualize his design as if it were live. 
Special goggles with little help from 
3D processing units can convert the 
workspace into a super- advanced 
arena of imagination where 3D draw- 
ings made in CAD software on com- 
puters come alive much the same way 
as Ironman designs his toys. 


Sports and Entertainment 

All work and no play makes AR look like a dull technology which it is not. 
AR can be as much fun as it is useful. You have already seen AR at work at 
several recent sports events, though this might not be obvious at first glance. 
A prime example is displaying the score and names of players directly on 
the field of play. AR also provides invaluable help in analyzing crucial 
event in real time, which in some sports can totally change the outcome of 
the game. Does not sounds familiar? Let us augment your knowledge with 
familiarity then; let’s talk about cricket. 

The LBW (Leg Before Wicket) is a well known method of dismissing a 
batsman, depending on (i) the trajectory of the ball and (ii) the object that 
the ball touches first in its trajectory. Instant action replays incorporating 
the AR technology called “Hawkeye” plot the trajectory of ball and predict 
whether the ball would have hit hit the stumps if it followed on its path 


NOTE 

I would like all the three pictures be 
printed as they together speak more 
than whatcan besaid via words 



AR in architecture can provide a high grade 
of imagination and aid to the architect 
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AR can have huge impacts on results of a game! 



without deviation. Another AR system known as “Hotspot” uses infrared 
cameras to highlight the point of contact between the ball and other surfaces 
based on show small deviations in temperature due to friction. 

Besides cricket, AR is also used in other sports such as in swimming to 
compare the current position of the lead swimmer with those just behind 
him or with a projection of where the world holder would be if he were in 
the same race. The sport of tennis has adapted AR technologies brilliantly to 
assist umpires and viewers in line calls, and football seems to be following 
closely with proposals by FIFA to introduce goal-line technologies. 

Advertisers are also utilizing AR to display ads in strategic positions on 
the field and having them blend with real-world elements. 

AR techniques are also being widely used in amusement park 
rides and stalls where children can play with virtual animals ranging 
from small puppies 
to giant dinosaurs. About 
a year ago, National Geo- 
graphic along with app- 
shaker demonstrated 
the possibility of AR in 
a shopping mall in the 
UK where people were 
amazed to see their own 
images but with wild 

animals and astronauts. AR can be very entertaining 
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This video should give you a hint of the same: http://www.youtube.com/ 
watch?v=TL62txWNFMY and we recommend you better be prepared 
to be awed. 

Education 

AR sure has a huge role to play in the future of education. Instead of 
teaching A for Apple and B for Ball’ to little children with picture books, 
AR can be used to visually project real-looking apples and balls in front 
them, and also have them experience these objects via other senses (touch, 
smell, sound and taste). Teachers could use applications which can simu- 
late chemical reactions, physics phenomena and maths concepts. The may 
“liven up” biology classes by creating virtual operation tables. AR could 
considerably reduce teacher. For example, a simulation of how various 
atoms and molecules would interact when brought close to each other 
would allow students to experiment and learn the concepts on their own. 
Rules could be understood faster and more easily even when a teacher 
is not present. 

Below is a list of a few interesting videos here to show how AR is going 
to revolutionize the way kids receive education in the future: 

► Chemistry lab simulation: http://www.youtube.com/watch?v=d8KXOrFLMvc 

► Yet another chemistry lab simulation: http://www.youtube.com/ 
watch?v=cPYKMhR50oO 

► Mathematics: http://www.youtube.com/watch?v=JKn7W8uAt3k 

► Biology: http://www.youtube.com/watch?v=6N8-x5WTJhA 

Art 

There is no better word to exemplify the uses of AR technologies in the 
field of art, than “Photoshop”. Adobe Photoshop, with the plethora of 
options and effects, is that special software that deserves recognition in 
IT heaven for turning many a nerd into capable artists. By superimposing 
images on top of each other, Photoshop allows new images and effects to 
be created. Detecting patterns and predicting their continuity in a scene 
can help turn boring things around us into pieces of art. Combined with 
dedicated and powerful hardware, AR could transform the sunset from 
your balcony into the most beautiful sunset on a beach! Mobile apps now 
let us “touch up” photos soon as we click them. The popular ‘negative’ 
and ‘sepia’ effects also qualify as means in which we have augmented our 
realities and produced pieces of art. 
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Machine-assisted translation 

AR’s role in tourism does not end with maps, directions and points of inter- 
ests. Wondering how else would a machine help you see a world differently 
when you are travelling? By assisting in translation, of course. Languages 
serve both as bonds and as boundaries when it comes to communication. 
When you travel far enough, bonds may easily turn into boundaries. For 
example, being able to understand your surroundings well, while you are 
in Spain would require you to understand Spanish. Billboards as well as 
restaurant menus may become intimidating in no time. Well, you do have a 
translation app or dictionary on your phone, but are you ready to translate 
all that you encounter into your preferred language? Well, good luck on 
your path to frustration and killing the joys of travelling. 

AR can provide immense help while you travel by translating any content 
that you encounter on the fly. There is no need to keep tapping in words 
to understand what a restaurant offers on its menu. All you need to do is 
to point your mobile phone’s camera at the menu and look at the display 
and you would be reading the menu in your preferred language in no time. 
This is achieved through several apps available in app stores, which almost 
instantly translate content captured through mobile phone cameras. Some 
of the really clever apps have been able to completely blur the distinguishing 
points the augmented and real-world features. 

One such application for iPhone is ‘Word Lens’ and we recommend you 
to watch it in action: http://www.youtube.com/watch?v=h20fQdTrHRs. We are 
sure you would love the innovative uses it can have. □ 
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FOR ILLUSION 
- WHAT MAKES 
AR TICK? 

Hardware and sensors are very critical 
in the implementation of Augmented 
Reality. Find out how... 
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A s you have probably figured out by now, Augmented Reality 
depends a great deal on the hardware. In fact you may even 
say it’s vital to its existence. In this chapter we will introduce 
you to the tools of the trade, the machinery which together will 
shape the way we will perceive our world in the future. 

The basic kit 

Although augmented reality has a lot of uses in various fields, they all have 
few similarities that cannot be ignored. They all need some sort of input 
which needs to be augmented. This input is fetched from the camera, most 
of the times. Next comes the part about deciding what graphical information 
needs to be shown on screen; depending on the application, the magnitude of 
annotations vary. However, in all cases some sensing devices are required to 
fetch information about what needs to be displayed. Depending on the type 
of interaction that the AR application needs to do with the user, it may need 
other input devices (for e.g. a tourism AR browser might feature a touch-to- 
learn feature). Next comes the dough of AR: a processor which combines 
the visual data being fed into the system with the processed changes that 
need to be displayed. Since we are in need of displaying the augmentation, 
AR also needs a display in most cases. 

One size does not fit all 

Depending on what the application is, in addition to the common hardware 
we mentioned above, it might need some more. For instance, the use of AR in 
tourism would need inputs from a GPS unit, an accelerometer and even a digital 
compass. On the other hand, usage of AR in military might require additional 
tagging and real world path highlighting. It might need information from custom 
devices, the local map and a few other information sources to assist the soldier. 
A virtual shopping trial room would work well enough with a webcam. In this 
chapter we take a look at the Hardware involved in AR, starting with display. 

Displays 

There are numerous ways to show the augmentation in AR systems and 
it is affected by the usage scope of the particular AR application. As per 
use-case the screen will differ. Let’s find out how. 

Head-mounted display (HMD) 

Head-mounted displays are the type of displays which can be mounted 
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(worn) on the head of a person. At times they are also called as ‘helmet 
mounted displays’. HMDs are of two types out of which one cannot be 
used for AR. 

1 HMDs that display only CGI - Most HMDs that you can find in the 
common market (though you will not be able to find them easily) are 
of this type and display only computer generated images (CGI). These 
HMDs are the ones which cannot be used for AR. The main require- 
ment of AR is to augment the reality. Since these HMDs do not feature a 
real-world view capability in the first place, the question of augmenting 
the view gets out of scope. These HMDs are usually used in virtual 
reality setups. 

2 HMDs that allow superimposition of CGI with real world view - 

These displays are capable of displaying real world imagery on which 
superimposition of additional data can be done. These once again are 
of two types: 

a Video see through: These HMDs take in the real view via a camera 
and mix the CGI infor- 
mation onto it. 
b Optical see through: 

These HMDs allow the 
real world to be viewed 
through a semi-reflec- 
tive mirror and project 
the CGI data on to the 
mirror using a projector 
device. The mirror 
being half reflective 
only creates the super- 
imposition effects. Yet 
another type of optical 
see through HMDs are 
the ones which use 
OLED displays which 
act as both viewing 
glass as well as displays. 

HMDs are one of the most 
useful type of displays although they are not the most common ones. The 
usage of HMDs are many. 



One of the various types of HMDs 
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1 Military: HMDs displays are built into fighter jet pilot’s helmet to displays 
important information about the flight and target. Often they have useful 
features like night vision and the visors may even be protective. HMDs 
are also used for the army and police where the displays show vital 


information about the surround- 
ings e.g. maps and thermal imaging 
data which can help detect a targets 
in very low light conditions. These 
HMDs also feature communica- 
tion gear for coordination among 
soldiers as well as for receiving 
commands from base. Most mili- 
tary applications of HMD include 
a OLED display instead of a semi- 
reflective mirror. 

2 Sports: Sports such as formula one 
racing can use AR with HMDs to 
show some important data about 
the race to the driver. 

HMDs feature head movement 
detection thus providing an intuitive 
and immersive experience to the user 
by giving them six degrees of freedom 


NOTE 

Degrees of freedom is a concept 
of mechanical engineering which 
determines what type of motions 
are allowed for an object. The 
degreesaredivided into two sets: 

1 Translational: Movement of 
theobjectfrom one pointto 
another. The three degrees 
under this consist of move- 
ments along the three axes 
of cartesian coordinates in 
space. 

2 Rotational: Rotation of object 
in the directions pointed by 
the axes in the cartesian 
coordinatesystem. 


Eye Glasses 

Eye glasses that are used in AR are essentially stripped down versions of 
HMDs. They do not feature all the tactical advantage of HMDs and the aug- 
mentation is simpler. In most of the cases, eye glasses used for AR purposes 
look just like the normal glasses with an additional camera. These devices 
allow the user to see the world as they would wearing the normal glasses 

and add the augmentation 
onto the view. They fea- 
ture OLED displays which 
allow the extra layer to be 
displayed directly on the 
glass. Some of them use 
the semi-reflective mirror 
approach as well. 



AR eye glasses look like the normal ones, though 
with a camera 
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Contact Lenses 

As strange as it might sound, contact 
lenses are already in development which 
would cover the eye with hundreds of 
LEDs to augment reality for you. These 
contact lenses are essentially tiny dis- 
plays which can be controlled by the 
built-in circuitry. As technology pro- 
gresses that tiny amount of real estate 
will even have integrated systems for 
wireless communications. There is still research going on in this area 
and the technology is only a little way past the drawing board. One of 
the ongoing research includes utilizing solar energy to power up all that 
electronics in the lens. 

Handheld displays 

If you are a smartphone user, chances are you have already used AR appli- 
cations. Even if you are not, we believe this is one AR displays which is 
not going to send any shockwaves of awe. Handheld displays are the most 
common AR displays and a lot of people already have them in in the form 
of smartphones. These displays possess nothing special in the hardware 
department from the AR viewpoint. 

It is the other hardware components of the phone which bring in the 
specialities. The components which aid AR on these displays mostly consists 
of a GPS module, accelerometer, digital compass and components used for 
data and communication. 

The disadvantage of handheld displays is that one needs to keep holding 
the device in front of oneself all the time. 

Spatial displays 

Technical speaking these displays are not the normal screens found in other 
devices. Instead the display is done using a projector. The projector projects 
the augmented view directly on the object of which the augmented view is 
needed. That being said, these setups (we would call them as ‘setups’ instead 
of ‘displays’ to save you some confusion) are not meant for individual users 
but for a group of users. 

Spatial displays have the advantage that they need not be worn or carried 
around (well, literally you cannot as they are preset for a given environment 



Contact lenses capable of doing AR 
are going to available in future 
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only) and since all users of the SAR (Spatial Augmented Reality) setup can 
see each other, they can easily collaborate. 

Though we have mentioned quite a number of displays that are used 
for AR, there are others which are not as popular. Let us now go to devices 
and methods used for inputting data into the AR systems. 

Input hardware for Interaction 

“We think basically you watch television to turn your brain off, and you 
work on your computer when you want to turn your brain on.” - Steve Jobs 

While opinions may vary greatly when it comes to Steve Jobs, it is pretty 
difficult to defy the logic contained in the sentence quoted above. Interaction 
is the main difference between two things that look just alike: a television 
and a PC. Both of them have got a display system but the former does not 
allow you to play as much with it as the latter does. While a TV would allow 
you to change channels and turn up or down the volume, a PC allows you 
to do much more (even while speaking of simply entertainment). 

AR is one of the most active areas of research in computing but strangely 
AR doesn’t require much interaction. In fact much of the interaction occur- 
ring within different types of AR implementations is automated to a level 
that you do not have to command the system once it has started working. 
A lot of research is directed towards minimizing whatever interaction you 
do with the system. For instance, if you were to wear a HMD, you would 
not want to keep pressing buttons to command the HMD to display what 
is on the right, instead you would turn right and the HMD would do the 
job of augmentation of the environment to your right. Such an experi- 
ence does not require you to constantly command the system. The system 
detects your movements and based on prediction of what you would want, 
changes the view. 

But not all AR implementations are used or are intended to be used 
without any interaction. Take for example the tourism AR applications 
you could install on your phone. Some of them do require you to tell them 
what information you want augmented into the real view. In the famous 
AR browser application ‘Layar’, you would have to first select the ‘layer’ 
of information which should provide the augmentation. Other methods of 
manual input involve verbal commands that can be given to an AR system. 

Let us again take the example of HMD. In most cases a HMD would 
cover the entire viewing area in front of your eye. In such a case it becomes 
very cumbersome to command the system using controls such as buttons. 
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Hence advanced HMDs must feature a system which allows them to com- 
mand the system using voice. But of course voice may not always work well 
in all AR systems. For example, if an architect is viewing a 2D drawing of 
a building in 3D using a HMD, commanding the HMD with voice to show 
minor changes in design will be a much more frustrating experience than 
using a wand or stylus to alter his view. 

Another way to send commands to an AR system is in the form a glove 
which detects the subject’s hand gestures and sends them to the processing 
unit which then takes an appropriate action based on that gesture. Yet 
another method to send manual input to an AR system would be touch. For 
example, an AR browser could be designed to show extra information about a 
building in front only if you touch the image on your mobile phone’s display. 

Input hardware for tracking changes 
in environment 

Before going further, we want to clear the point that all interaction hard- 
ware are input hardware in their own right. They allow the user to input 
commands to the system. What we are going to talk about in this part is 
not about them. In this part we talk about the hardware which allow the 
AR implementations to function automatically (and intuitively). Let’s start 
with the implementation of AR which we can encounter first hand - AR 
browser applications built for mobile phones. 

Use of AR on your mobile phone to assist you during your journey is 
one of the simplest usage of AR systems. Most AR browsers use a lot of 
information sources built into your phone to add information to your view 
of places around you. We have said earlier that AR browsers as such do not 
understand what they see. They show the information from the server (for 
the app) which depends on your phone sending its location and orientation. 
To learn the location, the AR browsers use the GPS module built into your 
mobile phone. If you are not getting GPS information, then the server for the 
application cannot know your exact location and AR would either not work 
at all or if it works, it would be inaccurate. The second piece of information 
it needs to know is which direction you are facing which is supplied by the 
compass built into the phone. The third source needed is what angle you are 
(your phone is) looking at from the ground. For example if you are trekking 
and you point your phone towards the base camp then it is important for 
the AR application to know that you were looking down or else it might 
mark the base camp as the hill’s peak. The information about the angle of 
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view is provided by the accelerometer in the device (the same thing which 
auto-rotates the pictures and videos as you turn your phone). Depending on 
the application, either the server or the phone calculates what the camera is 
seeing the then superimposes the information with the placemarks. 

HMDs, one of the prominent devices used for AR can facilitate quite a 
number of functions depending on what these are built for. 

HMDs which can be used 
by armed forces usually con- 
tain all components an AR 
browser for mobile would i.e. 
GPS, accelerometer and com- 
pass. The compass and accel- 
erometer in the HMD can be 
used to track head movements 
to provide a more immersive 
experience. Another addition 
can be that of eye tracking. If 
the computer has the sense of 
where the user is looking, it 
can improve the interface by adding more details to certain types of objects 
in the direction where the user is looking or help focus the user on only that 
area by causing other portions of screen to fade. However, in some cases 
the HMD does not have to contain a GPS such as when it is to be used in a 
closed environment and the position of the HMD is already known to other 
components of the AR setup. An engineer’s workspace could be an example. 

Additional input can be provided to the HMD using hand movements 
which can be done using special gloves which can be connected to the 
HMD using either wires or short range wireless methods (e.g. bluetooth). 
This enables gesture based input to the HMD for commanding the HMD 
without the use of voice. Gesture based input is used often for entertain- 
ment purposes wherein the user is allowed to play with virtual objects 
while hand tracking enables the user to interact with virtual objects. The 
day may not be far when you would be wearing a complete AR suit that 
adjusts what you see based on your blood pressure, skin humidity, muscle 
activity and other factors. 

Infrared sensors can be used to aid night vision as well as to see objects 
normally hidden from eyesight using clever use of colors. These can be used 
to show the presence of an object which might be perilous and save a life. 



Infrared inputs built into AR systems for cars can 
help prevent accidents 
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For example, use of infrared sensors in the windscreen of cars which can 
help drive safer in night by showing hot objects from a distance. This can 
include humans crossing the road. 

Despite so many input devices already present for AR, there are a number 
of methods being devised further to make the augmentation so intuitive that 
it blurs the lines between reality and augmented reality. 

Processing 

It is this part which actually does the work in any computing systems. 
No matter what computing system you pick up for the case study, it is the 
processing that would be at the center of it all. If everything works but the 
processing system is not working well, everything else - all input devices, 
tracking hardware and display systems - becomes useless. When it comes 
to AR, processing is more substantial and critical than other parts. Obvi- 
ously the processing units and methodologies are not simple at all. In fact 
they are sophisticated enough that this FastTrack is beyond the scope of 
any in-depth discussion on them. No matter, we will simplify the process 
for you in the next chapter and pique your curiosity a little more. TJ 
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THE MAGIC 
BEHIND THE 
SCIENCE 

We have answered the when, who, what 
and why of augmented reality. It’s time 
now to answer the how 


T he past few thousand words were dedicated to tell you about the 
hardware that’s used in AR. Before that we have been talking 
about the needs, significance and history of AR. We even talked 
about the different ways in which AR is implemented. It would 
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have become obvious to you that the technology is not simple. At various 
levels, we are in need of different techniques to achieve the goal. In the 
following few pages, we talk about the technologies that bind together 
the various parts to produce the mesmerizing effects of AR. We would go 
through each type one after another. 

Projection based AR 

We start off with projection based AR because the number of technologies 
and complexities of their interrelationships are the minimum here. In case 
of projection based AR, one has to deal with only a few parameters. The 
primary point of concern, as we have already said, is the creation of image 
sequence which is to be projected. Depending on the needs, quite a number 
of tools can be used. Let us begin with non-interactive AR. 

Non-interactive AR 

If the projection surface is just a plane surface, it is easy to generate the video 
sequence. The software required for the same would depend, once again on 
the specific need. If the need is only images, basic image processing tools 
along with video editing tools suffice. 

For more advanced uses which have captivating effects, more tools are 
required. Let us reconsider the case of projection on a cube with an edge- 
length of 2 meters. It is difficult to simply imagine a sequence which would 
look good on the cube without actually testing it. Let us not forget that the 
orientation of the cube matters - it may be kept on one of its sides or tilted 
on an edge or be placed on one of the corners. The variety of options depend 
on the projection surface and its complexity. 

In such cases a 3D modelling and animation software can prove 
immensely useful. Most such software can simulate any 3D environment 
and allow the designer to find out the best image sequences. They are also 
useful when experimentation space is not large enough. Since the software 
can produce animation with 3D objects, they can also help in generation 
of the actual light sequence to be projected. 3D animation software in non- 
interactive projection based AR can serve as a WYSIWYG tool for the result. 

Interactive AR 

In cases where projection based AR is used for interactive purposes, the 
target surfaces are mostly planar and the projection is not a light sequence 
that needs to be designed with a video editor or 3D animation tool well before 


tfVrtfrdidit .COfT 


56 


THE MAGIC BEHIND THE SCIENCE 


an event; instead it is produced via software in real-time. For the interaction 
to happen, at least a basic recognition system needs to be in place. The most 
important piece of technology which brings life to the augmentation depends 
on recognition systems, which we would learn about in a short while. 

Location based AR 

Next in line in terms of technical complexity is the location based AR. There 
are multiple factors at play. Location based AR apps take inputs from various 
sources. The information sources typically include: 

► GPS - It allows the app to pinpoint the location of the device. 

► Acceleration sensor - to find out how much you moved and in which 
direction. This prediction allows the app to update the view before 
the GPS data is received. Also, since accuracy of GPS is limited by 
approximately a couple of meters, it serves you when you move below 
that threshold. 

► Compass - To tell the direction you are looking in. 

► Orientation sensor - It tells the application how much the device is 
tilted. This helps predict the angle of view from the ground or in other 
terms, it helps figure out the location and angle of horizon relative to 
the device camera. 

The application helps locate points of interest on your screen by per- 
forming the following steps: 

1 Depending on the location received by the GPS system, the app requests 
its server to send a list of points-of-interest nearby along with their 
coordinates. 

2 It then reads the direction you are looking in (from the compass) and 
the angle at which you are looking. 

3 Combining these two pieces of information, it loads the data about the 
points-of-interest that it received from the server and overlays it on the 
camera view. 

May we remind again that most location based AR apps do not under- 
stand what they see - they do not recognize the buildings and mountains 
(or anything at all). All they do is to calculate your view relative to other 
points in your surroundings and place them on the screen. 

Trigonometry is what puts it all together. The role of AR application 
is simple - it has to place the information overlay at the correct places. 
The accuracy of a location based AR app depends on the accuracy of data 
received from different sensors. If the GPS data is not available, location 
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based AR apps cannot work properly. At the same time, an error in any 
other sensor would result in a shift of the augmentation layer (depending 
of what has gone kaput). 

Recognition based AR 

This is the most interesting breed on the list. The technologies that form the 
base here can aid other AR implementations. It also happens to be the most 
researched area for the same reason. Technologies involved in recognition 
based AR are as vivid as the use-cases. Recognition based AR depends 
on semantic understanding of the object in front of the camera and then 
replacing it with another object on the output display, or overlaying extra 
information related to the object. 

The multidimensional usage of recognition systems involve various 
disciplines and quite a significant part of that (if not all) is useful for AR. 
Before we start off, it would be good to keep in mind that not all technolo- 
gies pertaining to recognition systems apply to a single recognition based 
AR implementation. 

Image registration 

Image registration is a pro- 
cess of merging different sets 
of data of the same object into 
one image. The different data 
sets can be different images of 
the same object taken at dif- 
ferent times, from different 
angles and may include 
data received from other 
sources such as infrared 
view. When these different 
sets of data are merged into 
one, it may reveal a number 
of attributes which are not 
visible normally. 

The process of registra- 
tion involves more than one 
images. One of the images is 
called the target image and 



The process of Image Registration allows the faint 
mountain range in the background to be recognised 
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the other as reference image. During the registration process, the target 
image is transformed in a way that it aligns with the reference image. 
The transformation can be done in multiple ways, for each of them are 
best suited to certain types of images. The registration process can try to 
align target and reference images based on quite a number of registration 
methods. Some are: 

1 Intensity based: When entire portions of images can be matched. 

2 Feature based: When particular parts of images such as lines, contours 
etc can be matched. 

3 Transformations based: Rotated or scaled versions of images. 

4 Multi-modality based: When images produced by different types of 

sensors are joined together. 

The above list of registration 
methods is not exhaustive and there 
are many which are given a name by 
combining more than one of them, 
e.g. spatial registration method 
involves intensity, feature as well 
as transformation based methods 
at the same time. 

All these methods of image reg- 
istrations are imperfect. Though 
one of the prime concerns of image 
registration is to reduce errors. An 
error in matching the patterns in 
the received images can result in the 
target image overlaid on the refer- 
ence image with a planar difference 
between two points which represent 
the same part of image (it would be 
appear as if a displaced layer of the 
same image has been put over the 
actual one). 

Right from merging more than 
one images of the same object to showing more than one object on screen, 
this technology plays a role in recognition based AR. If the images received 
from more than one sensors are not merged properly, the application fails 
to serve its purpose. 


NOTE 

On these pages, by the word ‘image', 
we are not referring to just the ones 
which we take from a camera. An 
image produced by a radarwouLd 
not contain all the colors human eye 
can see but in terms of computing 
would still qualify as an image. 
Computers treat images as a map 
of pixels where each pixel contains 
some data. In case of a normal image 
(the ones we can take from a mobile 
camera), pixels contain color-data 
and in case of radar, pixels can be 
used to represent the distance of the 
point from the radar. Since such data 
can be expressed as a visible map, 
data received from a radarsystem 
can be used for image registration. 
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Object recognition 

It is very difficult for a person 
for look at a washing machine 
and confuse it with a tel- 
evision. Though humans can 
recognize a large number of 
objects easily, the recognition 

techniques are in their infancy Remember how Terminator recognised objects 

for computers. Object recogni- suc h as 9 uns ar| d even the jacket sizes 

tion has its roots in artificial 

intelligence, pattern recognition and is also one of the driving factors 
for the image registration process. Interestingly enough the technology 
has been imagined repeatedly by filmmakers, right from The Terminator 
to Transformers. 

Most object recognition methods heavily depend on objects that are 
already known. The larger the number of objects known to the system, 
better the chances of a correct object recognition. Let us first have a closer 
look at what recognition methods can act as hints in forming the complete 
understanding of the object. We would start with simple detection tech- 
niques and advance as we move forward. 

Looking for gradients 

Any curved surface reflects light as gradients as reflection varies in a con- 
tinuous fashion across the surface. The gradient produced by a cylindrical 
object is different from that of a conical object. Observing images received 
via the camera to study the gradients helps in figuring out the overall 
structure of the object. The variation of gradients based on the lighting 
conditions for each type of shape which build the object altogether pose a 
major challenge here. 

Comparing grayscale versions 

Grayscale versions of images are easier to compare than the colored ver- 
sions. Think of it like this: pictures of an pencil box when taken under 
green light looks different than picture of the same pencil box taken in blue 
light. But when you convert both the images into grayscale (i.e. black and 
white) the pictures do not look that much different. While to the human 
eye, the difference is trivial even in color, it makes a significant difference 
to a computer-based algorithm. 
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Looking for edges 

If you have used any sophisticated image editing program (such as Photo- 
shop, GIMP, Paint.NET etc), you would know of an effect which highlights 
edges in the pictures. A similar approach serves really well in detecting 

objects. Unlike gradients 
which vary under lighting 
conditions, edges do not. In 
most normal lighting condi- 
tions, location and size of 
edges do not change. 

Comparing the edges in 
the image received via the 
camera with entries in the 
database of images with sim- 
ilar edge patterns can reveal the nature of object. For example, the edge 
patterns of a table are going to be different than that of a chair. Change in 
the position of viewing angle however requires that the edge templates of 
known objects should also be available from various angles. 

Search in Large databases of object models 

This is more like an exhaustive search which would be done on a large set 
of object models. In this method, the image is submitted for a search in 
a database which contains the models for various objects from multiple 
angles. This method is synonymous to the exhaustive search method in 
other realms of data management and processing. 

Feature interpretation search 

If you aren’t clear with data structure concepts yet, this might not make 
much sense. In this method, first of all, features are searched for in the 
image e.g. search for a window’, a ‘door’ and a ‘chimney outlet’ in a picture 
of a ‘house’ should yield hits. Thereafter, comparisons are run for the 
identical placements of such features in the database of known objects. In 
our example, the object might just get recognized as a house. The search is 
done on the nodes of a feature tree. The root node of this tree is kept empty 
(indicating that no feature was matched). All child nodes in the tree contain 
all features found in the parent plus one more. As more and more shapes 
are found, the traversal goes down the tree to find the correct feature set 
identified in the image. 



The process of detecting Line of sight 
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It is notable that features can 
be built by nested patterns, e.g. A 
window can be rectangular in shape 
with two rectangular panes. If each 
pane contains a circle drawn over 
it for decorative purposes, then 
the windows can be identified as a 
rectangle made of two rectangles 
with a circle inside each. In such 
cases the successful recognition of 
the window would depend on the 
algorithm and the nodes available 
in the feature search tree. 

Yet another hurdle in this case 
is the size and relative position of 
other objects. Suppose that the fea- 
ture interpretation based search has 
confirmed that the object in front of 
the camera is a house but it also identifies a shoe in the image which happens 
to be larger than the house itself, then such a match would suggest that the 
house is actually a toy kept besides a real shoe. For the recognition algorithm 
to correctly peg this, it would require more than feature interpretation. 

Hypothesization and testing 

In this technique, various features are detected in the image first. The list 
of features are then searched against the list of features in known object 
models. The overall location of the features is also sent as a parameter 
for search. From the resulting set, the object model is then projected 
in software to decide whether the projection of the object returned as a 
search result would look similar to what is being seen by the camera. If 
the hypothetical image (projection of the object model received as a search 
result) is sufficiently similar to the image as seen by the camera, then the 
hypothesis is accepted and the object is recognised as the one received 
in search results. 

Scale Invariant Feature Transformation (SIFT) 

SIFT works by first storing key points (features) of an object and storing them 
in a database. This is done before SIFT is employed for object recognitions. 


WHAT IS A 
FEATURE? 

A feature, in terms of object 
recognition isa shape identifiable 
in a picture. Basic shapes (such asa 
rectangle, square, pentagon, circle 
etc.) too qualify as features. As you 
combinemorethan onefeature to- 
gether, you get more complex ones. 
In our case, a window isa feature and 
it is built by rectangles placed inside 
other rectangles. The identification 
of a feature depends on pattern 
recognition. can be used for image 
registration. 
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When the object has to be recognized, the features are detected along with 
their positions and alignments and searched in the database. 

Invariance 

This technique applies to those properties of of an object which are not 
subject to a change with the position of camera being changed. The condi- 
tion applies mostly to images of planar objects. 

There are a few other recognition techniques such as Geometric Hashing, 
Pose clustering, and Speed Up Robust Features (SURF). Unfortunately 
their understanding requires some complex mathematical theories which 
we cannot go in for the purposes of this FastTrack. 

Challenges of Object Recognition 

The biggest challenge in Object Recognition happens to be the wide array 
of possibilities available during the pattern search. In all the techniques we 
mentioned, a database with all the known object models is needed. However 
a database of object models of one technique do not serve well for others. 
For example, the database for searching through features in an image will 
be different from the one which uses grayscale images for comparison. 

Databases would again have to store the images of different objects from 
different angles in various formats for supporting more than one type of 
algorithm for pattern matching. While it increases the time taken for object 
recognition, it improves the accuracy and brings down the probability of 
error because none of the techniques are deemed perfect. 

Further difficulty is posed by the requirement to identify partly visible 
objects. Going back to our case of detecting a house - from some viewpoints, it 
is possible to achieve only limited visibility of the house - e.g. if you look at a 
house from the window of another house, you may get only limited visibility 
of the object. When the visibility of an object is obstructed, identification 
becomes more difficult. For this, the comparison algorithm sitting between 
the camera and object database has to be more clever. 

In many cases when more than one objects are present in front of the 
camera, one object would obstruct the view of another. Under such circum- 
stances, it is needed to detect both objects and figure out which object is 
obstructing the view of another. The simple method for such comparisons 
is to see which object provides a non-obstructing (complete) view and which 
one does not. Adding more objects in the mix may eventually bring out an 
erratic behavior from the algorithm. 
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Interpreting the video feed 

It is one thing to recognize an object and it is another to keep recognizing 
multiple objects multiple times every second when more can appear any 
time and current ones can disappear, move and rotate. Object recognition, 
although forms the base of the recognition based AR, it is still half the story 
told. The other half lies in interpreting the video feed. 

Videos are constructed by taking pictures at a very fast rate. For a video 
to be watchable, it should exceed the rate of 25 frames per second. Translate 
this requirement for AR application and it means that the application should 
not take more than l/25th of a second to identify objects. The creation of 
augmentation layer must also take place within the same time frame. 

The constraints call in for innovation of better hardware as well as 
more efficient algorithms at all levels. Ideally an algorithm using dedicated 
graphics hardware can do pattern recognition at a much faster rate than a 
completely CPU based algorithm. One method used for recognition based 
AR is to identify the object once and track its movement as it happens. The 
comparison of previous image frame with current one can suggest displace- 
ment and rotation of objects. In many cases a 3D wireframe model is present 
within the application to make the comparison which helps understand 
the differences better. 

Building 3D models 

By now, it would be obvious that a large part of recognition based AR 
implementation depends on the 3D models of objects. Building a sufficiently 
large database of object data is required to provide a better AR experience. 
While images are easier to insert (all you need to do is to take pictures), 
3D models can be difficult. One needs to design the model of the object 
before it could be fed into the database. Sure, there are tools available for 
the job. But the variation of design for even simple objects would lead to 
an enormous number of models. Modelling all of that into 3D is a daunting 
task. However with tools such as VideoTrace (i http://punchcard.com.au/ ), the 
amount of effort reduces. 

Together all these technologies and methods help in recognising the 
object that lies in front of the camera. Recognising the object is the core 
need of recognition based AR. The second part - overlaying the object with 
another object or presenting extra information about the object is easier. 
It involves creating a transparent layer on top of the actual view and then 
filling it with required information. 
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Detecting the line of vision 

This technique needs special kind of scanners or algorithm based cameras 
to track eye movement. It can be used for both augmented as well as virtual 
reality systems in various purposes. Detecting the line of vision can help 
relocate the augmentation layer on the right position. This is especially 
useful when the user is completely placed in the augmented space. This 
can include both HMDs and car windscreens. While in case of HMDs, the 

entire visible area can be augmented, 
a car windscreen would cover most of 
the visible space. Tracking the line of 
vision can help locate important infor- 
mation right in front of the eye rather 
than at a fixed location. 

So there you have it. To recap you 
have seen how the augmentation pro- 
cess takes place. The application pin- 
points where it is, considers the angle, 
orientation, direction and recognises 
what it’s looking at based on various 
techniques described above. It then overlays information about what it 
recognises to complete the augmentation process. Brilliant isn’t it? And 
what kind of wondrous applications can this result into? Check out the 
next chapter, f i 



The outline allows this object to be 
recognised by the algorithm as a car 
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30 BEST AR 
APPS FOR 
YOUR MOBILE 
DEVICE 

From translation utilities to apps that 
let you find your car - we cover the best 
Augmented Reality apps out there 


A ugmented Reality apps have been available for smartphones 
and tablets for quite a while now. By using your device’s 
camera, GPS and other sensors, the image through your camera 
is overlaid with additional bits of information. Below are a few 
AR apps that you’d like to check out. 
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1. Google Goggles 

Price: Free 

Platform: Android, iOS 4.0 

This amazing app from Google uses image recognition technology that lets 
you pull up information about various landmarks, logos, and texts, from 
the search engine mogul. The app lets you scan barcodes, add contacts via 
business cards, read book summaries by scanning the title, and returns 
information about pretty much anything - all by taking its picture. It can 
even translate the text, or look up the Metropolitan Museum of Arts’ website 
for information on famous artwork. The power of Google’s search engine 
combined with your camera! We don’t expect you to return someone’s wallet 
by recognizing a monastery, but it’ll definitely help you in all walks of life. 
And yes, it solves your Sudoku puzzles! 



2. Acrossair 

Price: Free 
Platform: iOS 

Acrossair is an AR browser for your iPhone and iPad that works as an awe- 
some 3D navigator to help you reach the place of your choice seamlessly. 
If you hold your device horizontally, it switches to Google Maps view to 
indicate your location. You can even pin your location to find your car. You 
can spin it around and view hotels, cinemas, landmarks and other geotagged 
entries in your vicinity. With integrated Facebook, Twitter and Flickr, you 
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can share pictures and location directly from your browser. After you’ve 
chosen the category of places you want to look up a.k.a layer, just hold your 
phone upright and you’ll see see-through image balloons floating around 
your screen overlaying information on the picture from your camera. 

3. iOnRoad Augmented Driving Lite 

Price: Free 

Platform: Android, iOS 

This app is your road guardian. Most smartphones would help you navi- 
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gate while on a road trip, but iOnRoad has so much more to it. It uses your 
phone’s camera, GPS and other sensors to prevent collision with vehicles 
in front of you by monitoring the headway distance. As soon as it detects 
that you’re moving too close to another car, or if you get off the road, it 
generates an audio-visual warning to help you take preventive action. It 
automatically detects that you’re parked and helps you get back to your 
car. It’s the ultimate on-road safety app; it was the best augmented reality 
app in 2011. So mount it on your windshield, and it’ll start automatically 
as you move. 

4. Google Sky Map 

Price: Free 
Platform: Android 

So you’re walking hand-in-hand with your girlfriend on a starry night, and 
she asks “Is that Ursa Major?”. Don’t be baffled my friend. Your guide to the 
galaxy is right there in your pocket. Just point your smartphone’s camera 
towards the sky, and this app will tell you all about that constellation in 
view. Move your camera around, and the stars will move with you. Not 
literally! Point to anything in the sky, and this app will tell you all about it. 
How about you learn a little by yourself first, and show it off later? 

5. Augmented Car Finder 

Price: Free 
Platform: iOS 

Yes it is very annoying when you are in a hurry, and you just can’t find 
your car in the parking lot. Every car seems the same at that time. This 
nifty app comes to your rescue. Like the name suggests, it’s a car finder. 
As long as you’re sober enough to use your i-device, this app will take 
you right to your car’s door. Wondering what’s “augmented” about it? 
When you launch this app, it uses your camera and gives you directions 
to where you placed your car. It also tells you the street address closest 
to your present location. 

6. AR Invaders 

Price: $1.99(iOS), $2.42(Android) 

Platform: iOS, Android 

This one’s for the Independence Day experience. Charge on like Will Smith 
and shoot the aliens. Everything around you turns into a warzone with this 
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app. Blast space invaders 
with your smartphones; 
stand up and fight in 360 
mode, or sit down for 180 
mode. You can even recruit 



an army with your friends 
in multiplayer mode. Stand 
up and fight for Earth, 
because these aliens are everywhere you point your smartphone. 


7. Theodolite 

Price: $3.99 
Platform: iOS 



Are you one of those 
people who want to 
know the altitude, tem- 
perature and other geo- 
graphical features of 
every place? You’re in 
luck! This app is great 
for adventurers, archi- 
tects, engineers and 
search and rescue guys. It gives you the topographical scenario of your 
surroundings, overlaying the information on the image from your smart- 
phone/tablet’s camera. It even lets you learn your elevation, inclination; it 
works as a compass, GPS, and a range-finder. So the next time you go hiking, 
this app will come in handy. 


8. Wikitude World Browser 

Price: Free 
Platform: 

Android, iOS, 
bada, BLackBerry, 

Windows Phone 

Wikitude is your 
smartphone’s AR 
encyclopedia. Use 
your smartphone/ 
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tablet’s camera, hold it up and look at your surroundings. You’ll see the 
world through it’s eyes. Get information on restaurants, landmarks and 
other places. Organize your “worlds”(content providers) according to your 
preference. Find ATMs, hotels or restaurants with your favourite cuisine 
from Yelp, Qype, Trip Advisor and Hotels.com. The Wikitude World Browser 
has been voted “Best Augmented Reality browser” three years in a row: 
2009, 2010 and 2011. 



9. Lookator 

Price: Free 
Platform: Android 

Launch this app, and see 
through the camera; you’ll see 
WiFi hotspots with their rela- 
tive strength and direction. The 
directions may not lead you to 
the hotspots, but it will defi- 
nitely get you a better signal. 


10. TagWhat 

Price: FREE 
Platform: iOS, Android 

Still carrying brochures for every 
place you visit in the holidays? 

No more flipping through pages 
looking up every landmark you 
encounter. TagWhat does all 
the work for you. It uses your 
smartphone’s camera and GPS, 
and pulls up information from 
the web and social networks about everything in your vicinity. It’s the 
ultimate tour guide. Read reviews, related stories and post your feedback. 
Sharing is what it’s all about. 

11. Satellite AR 

Price: Free 
Platform: Android 

You might not be an astronaut, but if you’re interested in knowing what 
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satellites are flying overhead, 
we have just the thing for 
you! Point the camera up in 
the sky, and little icons will 
show you all the satellites in 
the view. Just tap on the icon 
to know more about that par- 
ticular satellite. Who said you 
need to work at NASA to know about extraterrestrial objects? You just need 
your Android device. 

12. Layar 

Price: FREE 

Platform: iOS, Android, BlackBerry 

Layar is another AR 
browser for your smart- 
phone/tablet. So easy to 
find services near you any- 
time, anywhere. This app 
even displays tweets from 
nearby areas. Use your 
phone’s camera to scan any 
restaurant, building or even 
a corner around the street, 
and Layar will get you over 3000 layers of digital information to choose 
from. Scan your magazine and “make the print world clickable”. Save your 
favourites and share over the web. 

13. SpecTrek 

Price: $2.49 
Platform: Android 

Bored of sitting indoors with all 
your gadgets? Take this app out- 
side, and “hunt virtual ghosts in 
the real world!” SpecTrek is a fun 
app for children, that requires a lot 
of physical exercise. Scary ghosts 
appear in your surroundings(or in 
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the smartphone’s view of your place); you need to run towards them and 
shoot them with a net. The app provides bonuses, titles and records; basi- 
cally the drive to keep you moving. And if you’re a grown up but you don’t 
believe in growing up, here, go nuts! 


14. SpotCrime 

Price: $2.99 
Platform: iOS 

This app by spotcrime. 
com keeps track of all 
the crimes near you or 
anywhere else. It uses 
your iPhone’s GPS to 
tell you if you’re taking 
a safe street. It shows 
you criminal records 
/ occurences in the 
vicinity with different 
coloured icons. Point 
the camera towards a 
street, and check if it’s safe before you take the wrong route. It’s really 
useful in North America, but more areas are being covered worldwide. 




15. 3D Compass+ 

(AR Compass) 

Price: Free 
Platform: Android 

It’s a traveller’s tool. It 
has got navigation maps, 
a compass and local 
business finder apps all 
integrated into this one 
nifty application for your 
Android device. If that weren’t enough, it comes with an augmented reality 
view using your device’s camera. It overlays a compass on the image from 
the camera, and leads you in the right direction. There is also a map on your 
screen which helps you navigate. Turn up the brightness directly from it 
for better viewing in sunlight. 
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16. Spyglass 

Price: $4 
Platform: iOS 

This is another amazing augmented reality compass for your iPhone/iPad. 
It has got everything you’ll need if you’re ever lost in a jungle or hiking up 
a mountain-milspec compass, gyrocompass, speedometer, sextant, incli- 
nometer, angular calculator. It uses the GPS of your device to track your 



location in both geo and military formats. You can even save your location’s 
coordinates for coming back to it later. It’s even got an optical rangefinder 
that measures distance between you and an object in sight. Navigate by 
tracking positions of the Sun, the Moon and stars. Everything operates 
in an augmented reality view in 3D space at any orientation. A must have 
for adventurers. 

17. Augmented Colors 

Price: $0.99 

Platform: Windows Phone 7, 8 

Ever think of catching a butterfly in a jar, and to keep all the pretty colors? 
Now you can use your Windows Phone’s camera to pick and save colours. 
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This app is useful for art- 
ists and graphic designers. 

It identifies the RGB and 
hex values of a colour and 
stores the colour code to be 
used elsewhere. Just get the 
targeted colour between 
the crosshairs, and the 
screen and pin it on a live 
tile. A colour picker for 
real life! 

18. Yelp 

Price: Free 

Platform: Android, iOS 

Need help? Ask Yelp! Sounds cheesy, but it’s true. Yelp is your guide. Need 
a place to stay, or want to eat Thai food and you’re lost in the new city, Yelp 

will help you find anything any- 
where. Custom search allows you 
to narrow the choices by category- 
cuisine, distance, what’s open 
now. The Yelp Monocle uses aug- 
mented reality to overlay infor- 
mation on the image from your 
smartphone’s camera. Shake your 
iphone three times to launch this 
app. Browse through the reviews 
and ratings by fellow travellers 
and post your feedback. 

19. AugSatNav 

Price: Free 
Platform: Android 

This is the augmented reality 
version of Google Maps which 
overlays walking directions 
upon your smartphone’s camera feed of the path/road ahead. No need to 
zoom in and out of complicated maps, just follow the highlighted white 
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light on the screen 
and it’ll lead you to 
your destination. 
If you’re at cross- 
roads, and don’t 
know which side 
to go, the app will 
show an arrow to 
the right direction. 
No more asking 
strangers on road 
for directions. 


20. ARBasketball 

Price: Free 
Platform: iOS 

Bored at work? Now all you need is your 
i-device and a small piece of paper to 
play basketball. Well, it’s not as easy as 
throwing the ball into an actual hoop, 
but it’s pretty fun once you get a 
it’s technique, and addictive toe 
the marker on a white paper from http:// 
simiotica.com/arh_marker.pdj and point your 
phone’s camera at it to see a hoop. Swipe 
your finger up over the screen to throw the 
ball. An actual hoop doesn’t seem so cool 
now, does it? Another similar but amazing app, ARSoccer($2), let’s you 
play soccer. Just point your phone’s camera at the ground, and use your 
own feet to kick the virtual ball. 

21. Sun Seeker 

Price: $8.99 
Platform: iOS 

Sun-lovers, find the Sun anywhere, anytime! Yes, it may seem a little steep 
but it’s not just about the Sun’s location. The app shows you an augmented 
reality view of the solar path, it’s rise and set times, and a map view showing 
solar direction and elevations for each daylight hour of any day of the year. 
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This app is great for pho- 
tographers, gardeners, 
architects and even real- 
estate agents. Find the 
sun exposure of your 
property and optimal 
light conditions for your 
lobby and construct your 
house accordingly 

22. Paintball 

Price: Free 
Platform: Android 

Another amazing game 
on your Android device- 
real-time paintball 
in augmented reality. 

Choose from a variety of 
stunning paintball guns 
and play this awesome action-packed game against your friends. Just like the 
actual paintball game, this is also meant to be played with your friends. So 
connect via WiFi and “Ricochet, you take your aim... Fire away, fire away!” 
Specify the colour of your enemies t-shirt, play in well-lit areas and shake 
your phone to reload. Watch your back, your screen flashes and vibrates 
when you’re hit. 

23. Augment 

Price: Free 
Platform: Android 

Augment lets you visualise 3D 
objects such as paintings, books, 
furniture or even characters in 
your surroundings using your 
Android device’s camera. Fill your 
room with augmented objects. Try 
a new setting without actually 
rearranging everything. In order 
to use this app, print the marker 
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from http://augmentedev.com/markers.html. Then, select any 3D model from 
the gallery, aim your camera at the marker and voila! You will see the object 
in place of the marker on your device’s screen. Have fun creating your own 
3D characters and objects. 

24. GeoGoggle 

Price: Free 
Platform: Android 

This app lets you know 
the geographic details 
of your location: lati- 
tude, longitude, alti- 
tude in augmented 
reality. Also check your 
speed as well as the dis- 
tance to a specified location. It tells you directions using a 3D compass. Take 
photos while on the go, and use the map to see all the places where you clicked 
pictures. Use this app in conjunction with c:geo for mobile geocaching. Too 
many utilities, all packed into this one nift app with a user friendly interface. 

25. Kafkara Augmented Reality 

Price: Free 
Platform: Android 

Create talking avatars from your friends’ faces in augmented reality. Ava- 
tars can be created from the image from your camera. Create avatars from 
Facebook and Twitter profile pictures which then speak their latest status 
updates using text-to-speech technology. Interestingly, the app’s name is 
derived from the famous Franz Kafka novel The Metamorphosis’ where 
a man wakes up in the body of a beetle. Various bodies to choose from for 
your animated avatar. Now if you have a grudge against someone, or you 
just want to have fun, choose the funniest and creepiest of all the avatars. 

26. Junaio 

Price: Free 

Platform: Android, iOS 

Another augmented reality browser, it helps you locate theatres, restau- 
rants, taxi station all right in front of you. There’s also a barcode and QR 
code scanner in this app, so you get instant product information. Not as 
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many layers as in Layar, 
a good app nonetheless. 
It integrates various apps 
like Twitter, Wikipedia, 
Foursquare, Instagram 
to get updates on your 
favourite content all in 
one application. 

Price: Free 
Platform: Android 

This clever app simulates the vision of a color blind person. When you view 
an object through this app using the camera of your smartphone, it gener- 
ates two images: one that a normal person would see, and another version 
of it that an unfortunate color blind 
person would perceive. All common 
types of colour blindness are sup- 
ported- dyschromatopsia, protanopia 
(red color blind), deuteranopia (green 
colorblind), tritanopia (blue color 
blind) and more. You can test your 
own vision by comparing both the 
views. So if you have a friend who’s 
color blind, you’ll can see the world 
from his eyes. 

28. Nokia City Lens 

Price: Free 

Platform: Nokia Lumia Windows Phone 7 and higher 

This is the augmented reality browser for your Nokia smartphones with 
Windows Phone 7 or higher. You can download it from Nokia Beta Labs 
on your device. This is a great location based app, which helps you look up 
places in your surroundings by overlaying information on the camera feed 
of your smartphone. Now you needn’t look down upon your camera and use 
Nokia maps for navigation. All you need to do hold your phone upwards 
and see the magic! You can instantly see restaurants, bus stations, hotels; 
filter your search by category. It’s pretty, it’s useful, it’s to die for. 
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29. Peaks 

Price: $3 
Platform: iOS 

When you travel, you come 
across mountain ranges, ever 
wondered “Which mountain 
is that?” But no one seems to 
know the correct answer. The 
answer to these questions is 
here. This amazing app by 

Augmented Outdoors tells you everything about a mountain(major or 
minor). Just point your iPhone towards it, and it will do the rest. It’ll also tell 
you the mountain’s altitude and how far away it is from you. Click pictures 
with or without the overlaid information- double-touch a location bubble 
to make it disappear. On top of that, you don’t need an internet connection 
for this app to function. It works offline! Can it get any better? 
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30. DishPointer 

Price: $9.99 
Platform: iOS 

Just like the SatelliteAR for Android devices, we have DishPointer for 
iPhone/iPad users. Like always you have to pay a little extra for services 
on your i-device; but what you get, is just worth it. This beautiful app is the 
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ultimate satellite finder. Point your 
phone towards the sky and see on 
the live camera screen where the 
satellites are located. It was never 
easier to set up a satellite dish. 

Invaluable for campers. Adding 
to the list of amazing features, it 
does not even need cell reception 
or an internet connection. 

So far we’ve covered 30 cool AR apps that you should check out! There are 
many more such applications which are now switching over to augmented 
reality for a better and more beautiful user interface. The technology is 
moving on, and so should you! □ 
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CODING 
FOR AR - AN 
INTRODUCTION 

AR programming is neither just about 
mathematics nor just about code. There 
are hurdles and we are here to help you 
take a jump into creating AR apps 
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A R is magic to some, but magicians know that all magic is science. In 
this chapter we introduce you to the wands which build a magical 
layer on top of reality. 

Concepts and Domains Involved 

Augmented reality does not derive itself from a single engineering stream 
but many. It requires one to understand more than one technology at a 
time. Development of AR apps may require enormous amounts of effort 
depending on the type of development that needs be done. As much as it is 
true for any other type of application development, more critical the applica- 
tion, more testing and accuracy is required (a military application has to be 
much more fast, secure and precise than a traveller’s AR browser applica- 
tion). We have seen the types of augmented reality in previous chapters. In 
this one, we walk through the domain knowledge required by each of them. 

Location based AR 

Location based Augmented Reality is one of the easiest to program for 
obvious reasons - you would have all the resources required by you in the 
form of Android or iPhone (iOS) SDKs along with the device itself. The 
requirement here is to understand how to fetch data from sensors and use 
it to produce desired results. In most cases the APIs would provide you 
data in usable format but that would still be incomplete without in-depth 
understanding of what you are fetching. 

In addition to this, you are in need of understanding GPS system and its 
limitations. If you are unclear about the Latitude and Longitude systems, 



It is important to understand the rotes of Orientation Sensor and Accelerometer for creating 
a Location based ARapp 
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you need to brush up your high- school geography before you would be able 
to develop location based AR apps. 

Projection Based AR 

This one is more difficult to code for than location based applications. 
Projection based AR depends on three main factors - the surface on which 
projection has to be done, the light sequence to be projected and position of 
the projector. One point of projection based AR is that both development 
and implementation of an app is more expensive than a location based 
AR app. You are in need of an intelligent enough video editor (unless of 
course, you plan to project a static image), a powerful enough computer 
to help you with the job, a nice projector and a setup which allows you 
enough space to experiment. A few points which are worth a note about 
projection based AR: 

1 You will always need a low-light situation for projection based AR 
to work. 

2 You need enough space to develop the light sequence for projection, e.g. 
If you want to project your light sequence on a cube with an edge of 2 
meter, kept at a distance of 8 meters from the projector, then developing 
the light sequence by using a dice as a reference surface is a bad idea. 
Ideally, the target object’s size on which the simulation/testing has to be 
done should be the same as the target; but in case that cannot be done 
right away, the target object should still be large enough. It helps remove 
errors and allows better inspection during development. 

3 Projector resolution matters. You cannot use low resolution projectors 
built for PowerPoint presentations in a video sequence on a concert stage. 
When we talk about the domain of knowledge and experience required 

to undertake a projection based AR app, the skillset required is more 
mathematical than programmatic. Once again, it would depend on the 
specific application. 

1 If you are planning to use projection based AR in a concert, then the 
skillset required is going to be almost purely creative. There is hardly 
any programming involved except the tool which would help generate 
the light sequence. 

2 If you wanted to build a dialer on your palm, you would be in need of 
basic projection system (which would be probably mounted on your 
head) and a recognition system clever enough to understand what you 
are touching with your finger. In this case, you are more in need of image 
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recognition skills which again is a blend of various technologies and 
spans multiple research streams. 

Recognition based AR 

Recognition based AR depends heavily on pattern recognitions. More your 
app knows about patterns, more it can do. Pattern recognition technology 
has developed in leaps and bounds in the last half decade. Smile shot, face 
recognition for tagging and Google’s ‘Search by image’ feature are some of 
the most famous applications of image recognition technology. But this is 
nowhere close to completion. 

The first place from where you can start off with recognition based AR 
development is, once again, your smartphone because it already comes 
with the features of pattern recognition. Every time you scan a QR code 

to install an app, to open a webpage, you 
employ the pattern recognition faculties of 
your phone. But your phone can only get 
good enough for basic pattern recognition. 
If you want it to do advanced 3D object 
recognition, it might not prove to be the 
best gadget. 

3D object recognition requires much 
more data, processing power and efficient 
algorithms for getting the job done. While 

QR code scanning depends on data can p e stored on a phone and algo- 

pattern recognition . i . ! . ip * , 

rithms be designed tor the same, phones 

cannot have very powerful processors. Coming back to 3D object recognition, 

you would typically be in need of clever image recognition systems which 

know how an object appears from different angles. 

If you have to create an AR application which can recognise objects 

and then augment a layer on top of it, you are limited by some very basic 

shapes. For example, a popular development library AndAR (stands for 

Android Augmented Reality) which allows you to create recognition based 

AR applications for the Android platform supports only very basic pattern 

recognitions. Better computer vision and object recognition libraries are 

available for more powerful systems (such as for desktops and laptops) 

and with their aid, you can create more intelligent applications. It is easily 

predictable that in time, with more advanced hardware and software, such 

libraries will be available for mobiles as well. 
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From the knowledge domain perspective, recognition based AR requires 
deep understanding of computer vision and multimedia systems, especially 
that of video data (augmented reality works with continuous video feed 
from a camera; simple image recognition systems are not of much use here). 
Once again, all these disciplines depend on multiple mathematical research 
fields. If the library you are working with does not provide you with the 
required features, you are going to need superb mathematical skills to get 
things to work your way. 

Outlining based AR 

Outlining based AR heavily depends on object recognition and hence, a 
significant portion of knowledge required to work with recognition based 
systems is also applicable to the outlining based AR systems. However the 
game is not all the same. Outlining based AR applications may use data 
sources from other sensors too, such as compass, GPS, accelerometers, 
ultrasound sensors, infrared sensors and so on. Depending on the require- 
ments of the particular application you want to develop, conceptual and 
domain knowledge of one or more of the above will be necessary. 

Software required 

Let us say you want to build a location based AR browser. To achieve the feat, 
you need to plan first and for that, you would need answers to a few ques- 
tions. We try to show you some which you would most certainly have to face: 

► What platform should I build it for? 

If you are using an Android phone, you would target android; if you are an 
iPhone user, it’d be iOS. If you want to build a commercial application, you 
would want to target them both and may be others too, such as Blackberry 
and Windows Phones. The target platform depends on who your users are 
and need you wish to fulfil. 

► What programming languages can I use? 

That would depend on the platform that you are targeting. If you want 
Android, you need Java. Objective C if it’s for the iPhone while Windows 
and Blackberry phones allow you to get creative with multiple languages. 

► What OS can I use as a development platform? 

Once again, it depends on your target platform. Windows phone apps 
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need Windows for your development environment. Android apps can be 
developed on any platform where Java JDK works well. iOS development 
demands that you have a Macintosh with Xcode installed. 

► What IDE should I use? 

Windows Phone asks for Visual Studio and iOS asks for XCode. Android 
being based on Java, can do with any IDE as long as you are able to compile 
the sources and generate an APK package. Ideally Eclipse is the IDE used 
by most but other IDEs with more features (such as IntelliJ Idea) too sup- 
port Android SDKs. 

► How much help would I get on the web? 

iOS and Android cover majority of smartphone market and thus the help 
available on the web for one of these platforms is more than other platforms. 
Almost all platforms have well documented APIs which would help you 
as you progress. 

► How much time is it going to take? 

If you were to just throw the geographical coordinates on the user’s screen, 
it would not take much time (and that does not count as AR). But if you 
were to include a map and an object recognition system right into your AR 
browser app, then you are going to have your hands dirty with code for a 
longer time. It would also depend on what your experience has been as a 
developer for your target platform. 

As you can see, there are multiple variables in the mix and the most 
important factor which is going to affect your decision is the platform you 
want to target. Depending on the platform you want to build your app for, you 

have several restrictions. 
As a side note, one of the 
biggest reasons Android 
wins the apps battle is the 
freedom of options it pro- 
vides except in terms of 
programming language. 

In all cases nonethe- 
less, you are in need of the 
platform SDKs. In case of 
Android, you have a sepa- 
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rate NDK (Native Development Kit) which allows you to write native code 
for the target platform (remember that Android can be made to run on an 
Intel machine as well, while most mobile phones run on the ARM platform). 
The NDK compiles code to binary for the native platform. If you are going to 
program games which need OpenGL, you would need NDK in most cases 
to leverage the hardware capabilities. 

What frameworks do I have to aid me? 

It goes without saying that frameworks are built around a particular lan- 
guage (the only exception to this would be . A framework which supports 

multiple languages). You can neither use Codeigniter for creating a Java 
based app nor Qt framework for writing a Python script. We have already 
mentioned that the choice of language, IDE, development platform etc 
depend on the target platform. This translates to frameworks being mutu- 
ally exclusive for mobile platforms. In addition to this, frameworks which 
need to run on the mobile platform have to use the platform specific APIs. 
This shrinks the possibility to have cross-platform frameworks for AR uses. 
Here we list out the most popular and useful frameworks available to ease 
up the process of building AR apps. 

OpenCV 

A list of libraries which could help you do AR would be incomplete without 
OpenCV. OpenCV stands for Open Source Computer Vision and is one 
of the most famous libraries in the computer vision field. In case you are 
unaware, computer vision is the field of study which deals with any and 
every topic methods of acquiring, extracting, analyzing and understanding 
information from images to help a computer for take decisions in real world. 
Object recognition technologies come under the umbrella of computer 
vision. OpenCV claims to have more than 2500 optimised algorithms for 
computer vision of which, a majority can be useful in creating recognition 
based AR apps. 

OpenCV can be a great aid to AR programmers because of the language 
bindings that it offers. OpenCV has interfaces in C, C++ and Python as of 
now and soon, it would be available for Java as well. Since iOS apps need to 
be written in objective C, one can utilise OpenCV’s faculties by importing 
the library into the application. The fact that Windows phones can be pro- 
grammed in multiple languages is a benefit and OpenCV can be used with 
C++ for Windows phones. As far as Android is concerned, you can use the 


tfVrrirdidit .COfT 


88 


CODING FOR AR - AN INTRODUCTION 



Face detection with OpenCV library 


NDK (Native Development Kit) to cross compile OpenCV for android. An 
Android specific port is available at http://opencv.org/android. A blackberry 
port is also available for OpenCV. In case you want to build a desktop based 
application, OpenCV is compatible with Windows, Linux and Mac. 

The most attractive part of OpenCV is the efficiency. Being written in 
C++ with multithreading support, it runs fast. The API is organized and 
allows you to perform high level as well as low level functions on the data 
received. STL based interface is also a blessing. Despite being written in C/ 
C++ OpenCV manages the memory for you. This is a huge benefit for those 
who are afraid of pointers. OpenCV is licensed under a BSD license which 
means that it is both open source and free to use. You may use it for an open 
source project or for a commercial project. 

AndAR 

AndAR stands for Android Augmented Reality and - as evident from its 
name - is meant for the Android platform. The project seems a bit old since 
the official project page ( https://code.google.eom/p/andar/ ) shows the last release 
date back in 2010 implemented for Andoid Eclair and Froyo. Since majority 
of devices now run on Gingerbread which is mostly backward compatible 
with Froyo, this would still be helpful. AndAR uses the JNI (Java Native 
Interface) and contains code written in C (ah, the efficiency part) which 
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means that you would need the NDK, 
once again, to use it in your application. 

AndAR can only detect most 
basic shapes. You can find AndAR 
and AndAR Model Viewer apps in 
the android market and see them in 
action. The default shape that AndAR 
recognizes is a small L shape inside a 
box. Both the box and the shape inside 
should be of thick lines and black in 
color. If you wish to, you may create Default AndAR Marker 
your own markers. 

IN2AR 

IN2AR (pronounced “Into AR”) is the brainchild of a company called Beyond 
Reality located in the Netherlands and happens to be a SDK to help you 
build AR applications. The best part is IN2AD supports Unity. For those 
who think that Unity is just the UI of Ubuntu Linux, Unity is a game 
engine available for free at http://unity3d.com. The interesting part about 
Unity is its cross platform nature - it can help you reach 10 platforms: 
Desktop (Windows, Mac and Linux), Mobile (iOS, Android), Web (Pack- 
aged HTML pages, Flash) and Consoles (Wii, PS3 and XBOX 360). Since 
IN2AR supports unity, building your AR app in Unity will allow you to 
get cross platform publishing for your app easily enough so you can focus 
on the main functionality. IN2AR is written in Adobe ActionScript (AS3) 
and uses Adobe Native Extension Library for platform specific code. To 




IN2AR demo page shows the advanced capabilities it comes with 
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use IN2AR fully you’ll need to get the Adobe SDKs as well. The Flash SDK 
and AIR SDK are similar to each other while it is the Unity SDK which you 
might get interested in. Unity SDK comes with IN2AR plugins for iOS and 
Android (along with Windows and Mac). 

IN2AR is available in two licences - com- 
mercial and free. The free license has some 
restrictions such as added logo on web apps 
and 90 seconds operation on mobile apps. 
The commercial license is costly (around 
$2600 or Rs. 1,30,000). That is a huge sum if 
you only want to try and test the platform. If 
you have an awesome idea revolving around 
recognition based AR, you can start off with 
the free version. That said, the Unity engine 
too has a commercial license which you can 
avail for an amount (but well, the licensing 
is pretty flexible here). To understand why 
this framework in particular carries value, 
do visit the IN2AR demo page: http://www. 
in2ar. com/demo, asp 
Once you have gone through the demo, it is not difficult to see why IN2AR 
can be a good tool for creating apps like a virtual online dressing room. The 
Unity SDK makes sure that you can build games around Augmented Reality 
and publish it to not only mobile platforms but also to desktop gamers. Since 
IN2AR can detect multiple markers in a video feed, possibilities are high. 
Do note however that detection of 
more than one marker needs more 
processing and the performance 
would roll down. 

Layar 

Layar is one of the most famous AR 
location browsers which allows one 
to work with layers. You can create 
your very own layer which marks 
all the interesting outlets in the 
locality to help your friends. A food 
chain store can create one to help 


WORTH A TRY? 

The Videotrace tool allows you to 
extract out real life objects and 
build a 3D model of them. At the 
same time IN2AR allows easier 
detection of objects and create AR 
applications. Combining these two, 
it might just be possible to create a 
game/interactive app which merges 
real world with a virtual one. 



Unity can deliver on almost 
every platform 
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... Layer Vision recognizes 
the fingerprint,. 


Layar Vision recognizes the imagefingerprint 

users locate their outlets. A company can create one to help users locate 
their offices and so on. All with minimal effort. 

Although Layar has been an AR location browser, a relatively new tech- 
nology named “Layar Vision” from the company promises to bring a new 
dimension to advertising and to the print industry. It allows one to upload 
a picture or a set of pictures to their servers which together form part of a 
layer. When the user activates the corresponding layer, it readies his/her 
device for image detection. When the user points the camera towards the 
image in real world (with the Layar app running and the right layer loaded 
on the app), Layar Vision recognises the image and shows the content. The 
content may be an advertisement or promotional offer or a descriptive video 
or anything that the creator of the layer intended to show for the image. It 
takes the concept of QR codes and pushes it one step ahead. While QR codes 
can contain text and web links, Layar Vision can trigger a predefined action. 
Also, Layar Vision uses an AR layer so it is more interesting than looking 
at a non-interactive QR code. 

Layar APIs depend on REST technology which in turn depends on HTTP 
(just like majority of other applications). This translates to a lot of help 
being available on the web. The APIs are documented and you should not 
find much difficulty using them. Should you find yourself in trouble, quite 
a long list of posts and entries are available online along with discussion 
forums where others working on the platform could help you get through 
your niggles faster. 
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FLARToolkit 

FLARToolkit is based on ARToolkit. The development of original library 
(ARToolkit) appears to have stopped as the last release was back in 2007. 
However, FLARToolkit is still breathing and alive. The project is available 
online at http://www.libspark.org/ The website is in Japanese and the only 
way you can use it well enough is by using Google Translate. There is a 
discussion forum which is in Japanese too, unfortunately. The library has 
been utilised in quite a number of programs. If you are willing to use it, 
the official site may not be very useful (we assume you do not understand 
Japanese). However, help is available elsewhere by people who have used 
the library to create their own apps. 


Wikitude 

Wikitude is an AR based location browser app. As a developer it offers you 
SDKs for Android, iOS and Blackberry. You can develop apps in one of these 
or deploy using web technologies (HTML, JavaScript and CSS). There are 
multiple versions of the SDKs and you have to buy a license for the platform 
before you can publish the app. Once again, you have documentation and 
discussion forums to help you with. 
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Wikitude SDK is not free enough 


To sum it up... 

While there is no dearth of frameworks for almost any purpose, it is better 
to start-off with open source libraries. If you have not programmed an AR 
application before, the open source nature of the libraries would help you 
understand the underlying concepts, and guess what, you cannot skip the 
concepts even if you are using an AR library. 
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If your aren’t targeting a mobile platform, OpenCV is the best bet in 
the free world, and IN2AR if you are ready to pay. There are many other 
frameworks and libraries which we have not mentioned because they are 
either immature or their development has stopped long since. Also, before 
we end, we would like to emphasise on planning. Most AR applications are 
being targeted for mobile phones. If you too are going to develop for mobile 
phones, choose your target platform(s) wisely. Even more importantly, you 
should choose the framework which is known to work with the platform 
well enough. If the planned platform (and we do not just mean the OS, the 
device which you are going to test your application upon matters as well) 
does not support the technologies, amount of effort needed will rise. 

We wish you a very Happy New Tear ahead as we sign off. 
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